StrucPhysVideo: Learning Physical Dynamics from Structured Captions and Robot Actions

2026-09-16

AI Summary

研究通过结合结构化标注和机器人动作，使用StrucPhysVideo模型学习物理动态，以提高视频预测中物体运动、交互及状态变化的准确性。

Abstract

Modeling physical dynamics, including how objects move, interact, and change state, is central to video world models for embodied AI. We present StrucPhysVideo, a family of video world models that bridges physics-focused data curation with language- and action-conditioned prediction of scene evolution. Our data pipeline combines motion-aware video segmentation, quality and content filtering, and physical relevance verification with structured annotations of objects, materials, and temporally localized interactions. By disentangling camera motion from object behavior and explicitly describing contact, deformation, and state transitions, the pipeline provides supervision grounded in observable physical events. Building on these data, we introduce StrucPhysVideo-TI2V, a sparse Mixture-of-Experts (MoE) text-image-to-video model trained with a curriculum that progressively emphasizes physical dynamics while retaining general-domain video data. StrucPhysVideo-TI2V achieves state-of-the-art performance on Physics-IQ Verified, scoring 45.5% and outperforming Cosmos3-Super-Image2Video by 2.8 percentage points. Caption ablations across backbones further demonstrate the effectiveness of physics-focused supervision. We further extend StrucPhysVideo-TI2V to StrucPhysVideo-IA2V, an interactive image-action-to-video world model that predicts visual outcomes from robot end-effector commands. Action conditioning, causal autoregressive generation, and few-step distillation enable incremental robot rollouts with only four denoising steps. Together, StrucPhysVideo advances physical dynamics modeling from image- and language-conditioned video prediction toward action-driven interaction.

Problem

• Physical Dynamics
• Structured Captions
• Robot Actions
• Video World Models
• Embodied AI

Innovation

• Physics-focused Data Curation
• Language and Action Conditioned Prediction
• Sparse Mixture-of-Experts (MoE) Model
• Interactive Image-Action-to-Video World Model