VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotion

2025-02-03

AI Summary

To address the performance limitations of embodied robots in real-world visual navigation and gait control caused by the sim-to-real gap, this paper proposes a Real-to-Sim-to-Real paradigm. It constructs high-fidelity, physics-enabled digital twins from multi-view RGB images using 3D Gaussian Splatting, then integrates mesh-based dynamics simulation with RGB-only reinforcement learning to enable end-to-end training of navigation and gait policies under purely visual input. This approach is the first to jointly bridge both visual fidelity and dynamical consistency gaps, enabling direct policy deployment onto real quadrupedal platforms. Experiments demonstrate strong generalization and rapid adaptation in unseen, complex environments—including homes and factories—significantly improving real-world deployment efficiency.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Low Level & Physics-based Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Recent success in legged robot locomotion is attributed to the integration of reinforcement learning and physical simulators. However, these policies often encounter challenges when deployed in real-world environments due to sim-to-real gaps, as simulators typically fail to replicate visual realism and complex real-world geometry. Moreover, the lack of realistic visual rendering limits the ability of these policies to support high-level tasks requiring RGB-based perception like ego-centric navigation. This paper presents a Real-to-Sim-to-Real framework that generates photorealistic and physically interactive"digital twin"simulation environments for visual navigation and locomotion learning. Our approach leverages 3D Gaussian Splatting (3DGS) based scene reconstruction from multi-view images and integrates these environments into simulations that support ego-centric visual perception and mesh-based physical interactions. To demonstrate its effectiveness, we train a reinforcement learning policy within the simulator to perform a visual goal-tracking task. Extensive experiments show that our framework achieves RGB-only sim-to-real policy transfer. Additionally, our framework facilitates the rapid adaptation of robot policies with effective exploration capability in complex new environments, highlighting its potential for applications in households and factories.

Problem

• legged robots
• simulator-reality gap
• vision processing

Innovation

• VR-Robo
• Visual Navigation
• Sim-to-Real Adaptation