RoboScape: Physics-informed Embodied World Model

2025-06-29

AI Summary

Existing embodied world models exhibit deficiencies in modeling 3D geometry and motion dynamics, leading to physically implausible artifacts in video generation for contact-rich scenes. This paper proposes a physics-aware embodied world model that jointly learns RGB video generation and physics priors to implicitly encode object shape, material properties, and other physical attributes. Our core innovations include two physics-aware auxiliary tasks—temporal depth prediction and keypoint dynamics modeling—integrated with end-to-end neural rendering and explicit physical constraints in a unified training framework. This joint optimization significantly improves geometric consistency and motion plausibility. Experiments demonstrate that the model generates high-fidelity, physically realistic videos across diverse robotic manipulation scenarios, effectively supporting data-driven policy training and evaluation. Results validate its strong generalization capability and practical utility for embodied AI applications.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Intelligent Robots: Embodied AI
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

World models have become indispensable tools for embodied intelligence, serving as powerful simulators capable of generating realistic robotic videos while addressing critical data scarcity challenges. However, current embodied world models exhibit limited physical awareness, particularly in modeling 3D geometry and motion dynamics, resulting in unrealistic video generation for contact-rich robotic scenarios. In this paper, we present RoboScape, a unified physics-informed world model that jointly learns RGB video generation and physics knowledge within an integrated framework. We introduce two key physics-informed joint training tasks: temporal depth prediction that enhances 3D geometric consistency in video rendering, and keypoint dynamics learning that implicitly encodes physical properties (e.g., object shape and material characteristics) while improving complex motion modeling. Extensive experiments demonstrate that RoboScape generates videos with superior visual fidelity and physical plausibility across diverse robotic scenarios. We further validate its practical utility through downstream applications including robotic policy training with generated data and policy evaluation. Our work provides new insights for building efficient physics-informed world models to advance embodied intelligence research. The code is available at: https://github.com/tsinghua-fib-lab/RoboScape.

Problem

• Enhances physical awareness in robotic video generation
• Improves 3D geometry and motion dynamics modeling
• Integrates physics knowledge for realistic robotic scenarios

Innovation

• Unified physics-informed world model framework
• Temporal depth prediction for 3D consistency
• Keypoint dynamics learning for motion modeling