DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation

2026-08-13

AI Summary

This work addresses the challenge of maintaining action fidelity in robotic manipulation, where existing video prediction models often suffer from trajectory deviations or object disappearance. To tackle this, the authors propose an action-conditioned video world model that preserves the rigid-body structure of robotic arms through SE(3)-equivariant geometric encoding (PRoPE). The model integrates a lightweight depth branch, SAM³ segmentation masks, and a frozen V-JEPA teacher to enforce geometric and semantic consistency between actions and scene dynamics. Furthermore, a multi-step-to-few-step distribution matching distillation strategy is introduced to enhance deployment efficiency. The approach achieved first place in Track 1 and second place in Track 2 of the WorldArena 2.0 Challenge.

Abstract

We present \textbf{DreamX-Phi 1.0}, an action-conditioned video world model for robotic manipulation that, given an observed frame, a language instruction, and a prescribed action sequence comprising end-effector poses and gripper states, predicts the resulting future observations. Yet realism alone does not guarantee faithfulness: a convincing rollout can still move the wrong arm or lose the manipulated object. To ensure the prediction respects each arm's commanded path, we inject per-arm $\mathrm{SE}(3)$ transformations into attention via \textbf{PRoPE-style geometric encoding}, preserving arm identity and rigid-motion structure. Action control alone does not fully constrain scene geometry or the evolution of small manipulated objects. We therefore add a lightweight \textbf{depth branch} for scene-level geometry and use \textbf{SAM3 masks} with a frozen \textbf{V-JEPA teacher} to maintain object consistency throughout grasping. We further distill the multi-step generator into a few-step student via distribution-matching distillation for efficient deployment. At the time of writing, \model{} achieves first place on Track~1 and second place on Track~2 of the WorldArena~2.0 Challenge. Our model and code will be publicly available.

Problem

• video world model
• robotic manipulation
• action-conditioned prediction
• object consistency
• scene geometry

Innovation

• action-conditioned world model
• PRoPE-style geometric encoding
• SE(3) transformation
• object consistency
• distribution-matching distillation