Real2Sim: A Physics-driven and Editable Gaussian Splatting Framework for Autonomous Driving Scenes

2026-05-13

AI Summary

This work addresses the domain gap between real-world and simulated environments in autonomous driving scenario generation, where existing methods struggle to jointly ensure temporal/spatial consistency and physical plausibility. To overcome these limitations, we propose the first unified framework that integrates 4D Gaussian Splatting (4DGS) with differentiable Material Point Method (MPM), enabling high-fidelity, physics-driven, and editable reconstruction and synthesis of dynamic driving scenes. Our approach supports instance-level editing and realistic physical interactions, facilitating the generation of temporally coherent scenarios—including extreme cases such as collisions. Experiments on the Waymo Open Dataset demonstrate that our framework outperforms state-of-the-art methods in rendering quality, reconstruction accuracy, editing flexibility, and physical realism, thereby effectively supporting downstream tasks such as perception, trajectory prediction, and end-to-end policy learning.

Abstract

Reliable autonomous driving relies on large-scale, well-labeled data and robust models. However, manual data collection is resource-intensive, and traditional simulation suffers from a persistent reality gap. While recent generative frameworks and radiance-field methods improve visual fidelity, they still struggle with temporal and spatial consistency and cannot ensure physics-aware behavior, limiting their applicability to driving scenario generation. To address these challenges, we propose Real2Sim, an unified framework that combines 4D Gaussian Splatting (4DGS) with a differentiable Material Point Method (MPM) solver. Real2Sim explicitly reconstructs dynamic driving scenes as temporally continuous Gaussian primitives, supports instance-level editing, and simulates realistic object-object and object-environment interactions. This framework enables physics-aware, high-fidelity synthesis of diverse, editable scenarios, including challenging corner cases such as collisions and post-impact trajectories. Experiments on the Waymo Open Dataset validate Real2Sim's capabilities in rendering, reconstruction, editing, and physics simulation, demonstrating its potential as a scalable tool for data generation in downstream tasks such as perception, tracking, trajectory prediction, and end-to-end policy learning.

Problem

• autonomous driving
• simulation
• physics-aware
• scene generation
• reality gap

Innovation

• 4D Gaussian Splatting
• Differentiable MPM
• Physics-aware Simulation
• Editable Scene Reconstruction
• Autonomous Driving