Stable-Sim2Real: Exploring Simulation of Real-Captured 3D Data with Two-Stage Depth Diffusion

2025-07-31

AI Summary

To address the substantial domain gap between synthetic and real 3D data—hindering downstream real-world performance—this paper proposes a two-stage deep diffusion framework, enabling high-fidelity 3D simulation targeted specifically at locally distorted regions. Methodologically: (1) a residual learning strategy on paired depth maps focuses explicitly on structural error modeling; (2) a 3D-discriminator-guided loss enhances geometric consistency and texture realism; and (3) an end-to-end generation pipeline is built via fine-tuning Stable Diffusion. Contributions include: the first benchmark dedicated to evaluating 3D data simulation quality; and state-of-the-art improvements in realism, structural similarity (SSIM ↑12.6%), and downstream task performance—e.g., +8.3% mAP in 3D object detection.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Web Mining and Content Analysis: Web data generation and simulation
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

3D data simulation aims to bridge the gap between simulated and real-captured 3D data, which is a fundamental problem for real-world 3D visual tasks. Most 3D data simulation methods inject predefined physical priors but struggle to capture the full complexity of real data. An optimal approach involves learning an implicit mapping from synthetic to realistic data in a data-driven manner, but progress in this solution has met stagnation in recent studies. This work explores a new solution path of data-driven 3D simulation, called Stable-Sim2Real, based on a novel two-stage depth diffusion model. The initial stage finetunes Stable-Diffusion to generate the residual between the real and synthetic paired depth, producing a stable but coarse depth, where some local regions may deviate from realistic patterns. To enhance this, both the synthetic and initial output depth are fed into a second-stage diffusion, where diffusion loss is adjusted to prioritize these distinct areas identified by a 3D discriminator. We provide a new benchmark scheme to evaluate 3D data simulation methods. Extensive experiments show that training the network with the 3D simulated data derived from our method significantly enhances performance in real-world 3D visual tasks. Moreover, the evaluation demonstrates the high similarity between our 3D simulated data and real-captured patterns. Project page: https://mutianxu.github.io/stable-sim2real/.

Problem

• Bridging the gap between simulated and real-captured 3D data
• Overcoming limitations of predefined physical priors in 3D simulation
• Enhancing realism in 3D data simulation via two-stage diffusion

Innovation

• Two-stage depth diffusion model
• Finetunes Stable-Diffusion for residual generation
• Adjusted diffusion loss prioritizes distinct areas