High-Fidelity Simulated Data Generation for Real-World Zero-Shot Robotic Manipulation Learning with Gaussian Splatting

2025-10-12

AI Summary

Real-world robotic learning is hindered by the high cost of real-data collection and labor-intensive annotation, while existing simulation-based approaches suffer from visual and physical domain gaps that impede sim-to-real transfer. To address this, we propose Real2Sim2Real: a novel framework that first reconstructs high-fidelity 3D environments from multi-view real images, jointly leveraging 3D Gaussian splatting for appearance modeling and deformable mesh primitives for physics-aware interaction modeling. Second, it employs a multimodal large language model to autonomously parse scenes, infer object motion structures and physical properties, and generate drivable assets and simulation parameters. Finally, policies are trained exclusively in simulation and deployed zero-shot on real robots. Experiments demonstrate significant improvements over state-of-the-art methods across diverse real-world manipulation tasks, validating the framework’s strong generalization capability and practical efficacy.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Low Level & Physics-based Vision

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The scalability of robotic learning is fundamentally bottlenecked by the significant cost and labor of real-world data collection. While simulated data offers a scalable alternative, it often fails to generalize to the real world due to significant gaps in visual appearance, physical properties, and object interactions. To address this, we propose RoboSimGS, a novel Real2Sim2Real framework that converts multi-view real-world images into scalable, high-fidelity, and physically interactive simulation environments for robotic manipulation. Our approach reconstructs scenes using a hybrid representation: 3D Gaussian Splatting (3DGS) captures the photorealistic appearance of the environment, while mesh primitives for interactive objects ensure accurate physics simulation. Crucially, we pioneer the use of a Multi-modal Large Language Model (MLLM) to automate the creation of physically plausible, articulated assets. The MLLM analyzes visual data to infer not only physical properties (e.g., density, stiffness) but also complex kinematic structures (e.g., hinges, sliding rails) of objects. We demonstrate that policies trained entirely on data generated by RoboSimGS achieve successful zero-shot sim-to-real transfer across a diverse set of real-world manipulation tasks. Furthermore, data from RoboSimGS significantly enhances the performance and generalization capabilities of SOTA methods. Our results validate RoboSimGS as a powerful and scalable solution for bridging the sim-to-real gap.

Problem

• Bridging the sim-to-real gap in robotic manipulation learning
• Automating creation of physically plausible articulated assets
• Generating scalable high-fidelity simulated data from real images

Innovation

• Converts real images into interactive simulation environments
• Uses Gaussian Splatting for photorealistic scene reconstruction
• Employs MLLM to automate physical property inference