GASE: Gaussian Splatting-Based Automated System for Reconstructing Embodied-Simulation Environments

2026-06-16

AI Summary

Existing methods for constructing simulation environments suffer from low data acquisition efficiency and insufficient accuracy in foreground object extraction, hindering the generation of high-fidelity training scenes with minimal sim-to-real gaps. This work proposes a highly automated simulation scene construction system that captures multi-view videos using a panoramic camera array and leverages camera pose information to achieve robust cross-frame 2D foreground extraction. It further reconstructs static backgrounds via high-quality image inpainting and seamlessly integrates 3D Gaussian Splatting with a physics simulator for end-to-end scene generation. The proposed approach surpasses existing 3D Gaussian-based methods by over 10% in object segmentation accuracy and achieves state-of-the-art background inpainting quality. Policies trained using this system exhibit performance on real-world robotic manipulation and navigation tasks within 10% of those trained exclusively on real data.

Abstract

Training embodied agents in the real world requires skilled operators and expensive hardware. Simulation environments offer a compelling alternative by enabling large-scale, cost-effective data augmentation. Consequently, rapidly constructing high-fidelity simulation scenes with a minimal sim-to-real gap has become a critical objective in robot learning. While reconstruction-based methods provide superior visual quality, current workflows are hindered by inefficient data acquisition and subpar foreground object extraction. We thus propose GASE, a highly automated system for simulation scene construction. GASE leverages multi-view video streams from panoramic camera arrays to enable rapid environment scanning. To ensure high-quality asset generation, our pipeline introduces a camera-pose-based strategy that robustly extracts objects across frames in the 2D domain, followed by high-fidelity scene inpainting. Foreground objects and the static background are then reconstructed independently and seamlessly imported into physics simulators for policy training. Extensive experiments demonstrate that GASE outperforms existing 3D Gaussian-based methods in segmentation accuracy by over 10\% while achieving state-of-the-art inpainting quality. Furthermore, real-robot deployments across manipulation and navigation tasks maintains a performance gap of less than 10\% compared to policies trained purely on real-world data. These results confirm that GASE provides an efficient and highly effective solution for bridging the sim-to-real gap. Code will be released.

Problem

• sim-to-real gap
• embodied agents
• simulation environments
• 3D reconstruction
• foreground object extraction

Innovation

• Gaussian Splatting
• sim-to-real gap
• automated scene reconstruction
• camera-pose-based segmentation
• embodied simulation