PhysMani: Physics-principled 3D World Model for Dynamic Object Manipulation

2026-07-02

AI Summary

Manipulating fast-moving dynamic objects in unstructured 3D environments remains highly challenging, as existing vision-language-action and world model approaches struggle to accurately capture 3D geometry and generate physically plausible future states. This work proposes a novel framework that integrates physical priors into a 3D Gaussian world model coupled with a forward-looking policy architecture. It introduces, for the first time, a divergence-free Gaussian velocity field, optimized online to ensure physically consistent dynamics prediction. Furthermore, a cross-attention mechanism with learnable tokens seamlessly incorporates these predicted dynamics into a unified vision-language-action policy. The authors also introduce PhysMani-Bench, a new benchmark comprising 16 diverse tasks. Experiments demonstrate substantial improvements over strong baselines, with significantly higher task success rates in both simulation and real-world robotic settings.

Abstract

Manipulating fast and dynamically moving targets in unstructured 3D environments remains challenging for embodied AI. Existing visual-language-action models and world models struggle with accurate 3D geometry and physically meaningful forecasting. We propose PhysMani, a framework that couples a physics-principled 3D Gaussian world model with a future-aware action policy model. The world model learns a divergence-free Gaussian velocity field via online optimization for fast and physically grounded future dynamics prediction. The policy model integrates the predicted 3D scene future dynamics through a learnable token based cross-attention module. We introduce PhysMani-Bench, a dynamic manipulation benchmark with 16 tasks, and demonstrate a superior success rate over strong baselines in both simulation and real-world robot experiments.

Problem

• dynamic object manipulation
• 3D world model
• embodied AI
• physically meaningful forecasting
• unstructured 3D environments

Innovation

• physics-principled world model
• 3D Gaussian velocity field
• future-aware policy
• dynamic object manipulation
• divergence-free dynamics