Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin for Real-World Robot Policy Evaluation

2025-04-04

AI Summary

To address the high cost and low frequency of real-world policy evaluation in behavioral cloning, this paper proposes an end-to-end framework integrating dynamic digital twins based on Embodied Gaussians, spanning real-world data collection, offline parallel training in simulation, and co-deployment across virtual and physical environments. It pioneers deep integration of dynamic digital twins throughout the entire behavioral cloning lifecycle and achieves, for the first time, robot joint-level real-time vision-kinematic closed-loop simulation—decoupling policy execution from hardware constraints and significantly mitigating sim-to-real transfer challenges. Leveraging multi-view neural rendering and state-action joint simulation alignment, the method ensures strong consistency between simulated and real-world task success rates (demonstrated with statistically significant correlation on the PushT benchmark). Consequently, it drastically reduces reliance on costly real-world evaluation, enhances policy screening efficiency, and improves detection of overfitting and underfitting.

Technology Category

• Intelligent Robots: Embodied AI
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Recent advancements in behavior cloning have enabled robots to perform complex manipulation tasks. However, accurately assessing training performance remains challenging, particularly for real-world applications, as behavior cloning losses often correlate poorly with actual task success. Consequently, researchers resort to success rate metrics derived from costly and time-consuming real-world evaluations, making the identification of optimal policies and detection of overfitting or underfitting impractical. To address these issues, we propose real-is-sim, a novel behavior cloning framework that incorporates a dynamic digital twin (based on Embodied Gaussians) throughout the entire policy development pipeline: data collection, training, and deployment. By continuously aligning the simulated world with the physical world, demonstrations can be collected in the real world with states extracted from the simulator. The simulator enables flexible state representations by rendering image inputs from any viewpoint or extracting low-level state information from objects embodied within the scene. During training, policies can be directly evaluated within the simulator in an offline and highly parallelizable manner. Finally, during deployment, policies are run within the simulator where the real robot directly tracks the simulated robot's joints, effectively decoupling policy execution from real hardware and mitigating traditional domain-transfer challenges. We validate real-is-sim on the PushT manipulation task, demonstrating strong correlation between success rates obtained in the simulator and real-world evaluations. Videos of our system can be found at https://realissim.rai-inst.com.

Problem

• Bridging sim-to-real gap for robot policy evaluation
• Reducing costly real-world evaluations with digital twin
• Aligning simulated and physical worlds for accurate training

Innovation

• Dynamic digital twin aligns simulation with reality
• Embodied Gaussians enable flexible state representations
• Simulator decouples policy execution from hardware