ChronoDreamer: Action-Conditioned World Model as an Online Simulator for Robotic Planning

2025-12-21

AI Summary

This work addresses robotic manipulation planning in contact-rich environments by proposing an action-conditional world model that jointly predicts future video frames, contact distributions, and joint angles. Methodologically: (1) it introduces depth-weighted Gaussian splatting to encode contact information, enabling high-fidelity contact modeling; (2) it integrates spatiotemporal Transformers with MaskGIT-style masked modeling to support multimodal, joint prediction; and (3) it incorporates a vision-language model (VLM) as a collision-aware trajectory rejection sampler to enhance planning safety. Evaluated on DreamerBench, the model significantly improves the plausibility of contact predictions and spatial consistency of non-contact motion. The VLM-based rejection sampler accurately discriminates between colliding and collision-free trajectories, thereby substantially increasing planning robustness and safety.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Manipulation
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We present ChronoDreamer, an action-conditioned world model for contact-rich robotic manipulation. Given a history of egocentric RGB frames, contact maps, actions, and joint states, ChronoDreamer predicts future video frames, contact distributions, and joint angles via a spatial-temporal transformer trained with MaskGIT-style masked prediction. Contact is encoded as depth-weighted Gaussian splat images that render 3D forces into a camera-aligned format suitable for vision backbones. At inference, predicted rollouts are evaluated by a vision-language model that reasons about collision likelihood, enabling rejection sampling of unsafe actions before execution. We train and evaluate on DreamerBench, a simulation dataset generated with Project Chrono that provides synchronized RGB, contact splat, proprioception, and physics annotations across rigid and deformable object scenarios. Qualitative results demonstrate that the model preserves spatial coherence during non-contact motion and generates plausible contact predictions, while the LLM-based judge distinguishes collision from non-collision trajectories.

Problem

• Develops a world model for robotic manipulation predicting future visual and contact states.
• Uses a vision-language model to assess and filter unsafe actions before execution.
• Evaluates the model on a simulation dataset with rigid and deformable object scenarios.

Innovation

• Action-conditioned world model predicts future frames and contacts
• Contact encoded as depth-weighted Gaussian splat images
• Vision-language model evaluates rollouts for collision likelihood