RoboDream: Compositional World Models for Scalable Robot Data Synthesis

2026-06-01

AI Summary

This work addresses the high cost and limited scalability of real-world robot demonstration data, as well as the insufficient physical plausibility and diversity of existing synthetic data generation methods. The authors propose a robot-centric, generalizable world model that decouples trajectory execution from environment synthesis, enabling high-quality manipulation data generation across novel objects, scenes, and viewpoints. They introduce two innovative data augmentation paradigms—"retrieve-and-regenerate" and "prop-free teleoperation"—which reuse existing motion trajectories to automatically synthesize corresponding target objects and environments without requiring additional robot demonstrations. By integrating video diffusion models, robot motion rendering, and explicit priors on scenes and objects, the framework achieves controllable and physically consistent data generation. Experiments demonstrate that the synthesized data significantly improves downstream policy performance while substantially reducing reliance on real-world data across diverse manipulation tasks.

Abstract

Scaling robot learning requires large-scale, diverse demonstrations, yet real-world data collection via teleoperation remains prohibitively expensive and time-consuming. While video diffusion models offer a promising avenue for data scaling, existing generative approaches are often limited to superficial visual augmentation, or suffer from embodiment hallucinations that yield physically infeasible motions. We present a generalizable embodiment-centric world model that achieves scalable data generation by synthesizing photorealistic demonstrations with novel objects, in novel scenes, and from novel viewpoints. Our approach anchors generation to rendered robot motion while conditioning on explicit scene and object priors, effectively decoupling trajectory execution from environment synthesis. This formulation has the potential to unlock two powerful data scaling capabilities: (1) retrieval and rebirth, which repurposes existing trajectories into entirely new contexts without new motion data; and (2) prop-free teleoperation, where operators manipulate empty air and the model hallucinates the target objects and scene afterwards, eliminating reset time. We demonstrate with real-world experiments that our generated data consistently improves downstream policy performance and significantly reduces real-world data requirements across diverse manipulation tasks.

Problem

• robot data synthesis
• scalable robot learning
• embodiment hallucination
• photorealistic demonstration
• world model

Innovation

• compositional world models
• embodiment-centric generation
• data synthesis
• prop-free teleoperation
• trajectory-environment decoupling