ComSim: Building Scalable Real-World Robot Data Generation via Compositional Simulation

2026-04-13

AI Summary

This work addresses the challenges of scarce real-world training data, limited environmental diversity, and high human annotation costs in robotics by proposing Compositional Simulation—a novel hybrid framework that integratively combines classical and neural simulation paradigms. The approach establishes a closed-loop “real-to-sim-to-real” data augmentation pipeline, leveraging minimal real-world data to generate large-scale, high-fidelity training sets that span a broad spectrum of realistic scenarios. By preserving physical consistency while substantially narrowing the sim-to-real domain gap, the method significantly enhances the success rate of learned policies when deployed in physical environments. Experimental results demonstrate that the proposed framework offers an efficient and scalable data generation paradigm for robot learning, effectively bridging the divide between simulation and reality.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Web Mining and Content Analysis: Web data generation and simulation
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Recent advancements in foundational models, such as large language models and world models, have greatly enhanced the capabilities of robotics, enabling robots to autonomously perform complex tasks. However, acquiring large-scale, high-quality training data for robotics remains a challenge, as it often requires substantial manual effort and is limited in its coverage of diverse real-world environments. To address this, we propose a novel hybrid approach called Compositional Simulation, which combines classical simulation and neural simulation to generate accurate action-video pairs while maintaining real-world consistency. Our approach utilizes a closed-loop real-sim-real data augmentation pipeline, leveraging a small amount of real-world data to generate diverse, large-scale training datasets that cover a broader spectrum of real-world scenarios. We train a neural simulator to transform classical simulation videos into real-world representations, improving the accuracy of policy models trained in real-world environments. Through extensive experiments, we demonstrate that our method significantly reduces the sim2real domain gap, resulting in higher success rates in real-world policy model training. Our approach offers a scalable solution for generating robust training data and bridging the gap between simulated and real-world robotics.

Problem

• robot data generation
• sim2real gap
• scalable training data
• real-world consistency
• data scarcity

Innovation

• Compositional Simulation
• sim2real
• neural simulator
• data augmentation
• robotics