RoboSnap: One-Shot Real-to-Sim Scene Generation for Generalizable Robot Learning and Evaluation

2026-07-07

AI Summary

Constructing physically stable and visually realistic simulation environments is costly, hindering the generalization and reproducible evaluation of robotic policies. To address this challenge, this work proposes RoboSnap, a framework that generates interactive, simulation-ready scenes from a single RGB image. The approach employs a hierarchical design that decouples physically interactive regions from visual backgrounds: foreground objects are represented with collision-aware assets to ensure dynamic plausibility, while the background leverages 3D Gaussian splatting to preserve multi-view photorealism. The authors introduce the DROID-Sim dataset and demonstrate the framework’s effectiveness on both DROID and real-robot tasks, showing strong performance in trajectory replay, task-oriented data generation, and preserving sim-to-real policy evaluation fidelity.

Abstract

Recovering real-world scenes as interactive simulation environments can enable generalizable robot learning and reproducible policy evaluation. However, constructing scenes that are both physically stable and visually faithful remains slow and expensive. In this work, we present RoboSnap, a real-to-sim framework that turns a single RGB image into a simulation-ready scene. The key idea is a layered design that separates the physics-critical interaction area from the surrounding visual context: collision-aware foreground assets are refined for stable robot interaction, while a 3D Gaussian splatting visual layer preserves faithful background appearance under novel views. Experiments on DROID scenes and real-robot tasks show that RoboSnap achieves reliable trajectory replay in the recovered scenes, supports task-specific synthetic data generation for policy training, and yields meaningful sim-real correlation for policy evaluation. To further support real-to-sim research, we introduce DROID-Sim, a real-to-sim companion dataset constructed from 564 real-world scenes in DROID. Extensive experiments suggest that the value of real-to-sim methods lies not only in high-fidelity visual reconstruction, but in turning real environments into reusable infrastructure for robot learning and evaluation.

Problem

• real-to-sim
• scene generation
• robot learning
• simulation
• policy evaluation

Innovation

• real-to-sim
• one-shot scene generation
• layered simulation design
• 3D Gaussian splatting
• robot learning