DISCOVERSE: Efficient Robot Simulation in Complex High-Fidelity Environments

2025-07-29

AI Summary

To address the performance gap in Sim2Real transfer caused by insufficient fidelity and physical consistency in existing simulators, this paper introduces the first unified, modular, open-source simulation framework integrating 3D Gaussian Splatting (3DGS) with MuJoCo. The framework establishes an end-to-end Real2Sim2Real closed loop: a full-pipeline Real2Sim workflow converts real-world scenes into photorealistic geometric and appearance reconstructions; concurrently simulates multi-modal sensor data (RGB, depth, IMU); and performs high-fidelity rigid-body dynamics simulation—while maintaining compatibility with existing 3D assets, robot models, and the ROS ecosystem. Its core innovation lies in the first deep coupling of 3DGS’s neural rendering capability with MuJoCo’s physics engine, enabling joint geometric-appearance-dynamic modeling and substantially improving cross-domain generalization. In imitation learning tasks, it achieves state-of-the-art zero-shot Sim2Real transfer performance and significantly enhances efficiency in robot policy training and evaluation under complex environments.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Multi-modal Vision

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

We present the first unified, modular, open-source 3DGS-based simulation framework for Real2Sim2Real robot learning. It features a holistic Real2Sim pipeline that synthesizes hyper-realistic geometry and appearance of complex real-world scenarios, paving the way for analyzing and bridging the Sim2Real gap. Powered by Gaussian Splatting and MuJoCo, Discoverse enables massively parallel simulation of multiple sensor modalities and accurate physics, with inclusive supports for existing 3D assets, robot models, and ROS plugins, empowering large-scale robot learning and complex robotic benchmarks. Through extensive experiments on imitation learning, Discoverse demonstrates state-of-the-art zero-shot Sim2Real transfer performance compared to existing simulators. For code and demos: https://air-discoverse.github.io/.

Problem

• Develops a unified 3DGS-based simulation framework for robot learning
• Bridges Sim2Real gap with hyper-realistic geometry and appearance synthesis
• Enables parallel multi-sensor simulation and accurate physics for robotics

Innovation

• Unified modular open-source 3DGS simulation framework
• Gaussian Splatting and MuJoCo for parallel simulation
• Hyper-realistic geometry and appearance synthesis pipeline