Cam2Sim: Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation

2026-07-06

AI Summary

This work addresses the visual and behavioral discrepancies between synthetic imagery and real-world observations in autonomous driving simulation by proposing a novel method to transform real driving videos into closed-loop CARLA simulation scenarios. The approach reconstructs road geometry, ego-vehicle trajectories, static vehicles, and environmental assets, and—integrating Gaussian splatting rendering into the simulation pipeline for the first time—achieves high-fidelity image generation. The framework supports an end-to-end pipeline from ROS data extraction and OpenStreetMap-based map generation to CARLA scene construction. Experimental results in real urban environments demonstrate that the proposed method significantly enhances photorealism compared to conventional rendering techniques and improves the consistency between simulated closed-loop driving behavior and real-world trajectories.

Abstract

Simulation-based testing enables safe and repeatable evaluation of autonomous driving systems, but its effectiveness is limited by the gap between synthetic simulator outputs and real-world camera observations. To address this problem, we present Cam2Sim, a tool that transforms real-world driving recordings into playable CARLA simulation scenarios. Starting from camera images and poses, Cam2Sim reconstructs road geometry, ego trajectories, parked vehicles, and simulation assets, and augments the reconstructed environment with Gaussian Splatting to render camera observations that resemble the original recording. The framework supports ROS-based data extraction, parked-vehicle detection, OpenStreetMap-based map generation, CARLA scenario construction, Gaussian Splatting training, trajectory replay, and closed-loop execution with a system under test. We validate Cam2Sim on a real-world urban-driving scenario with a camera-based end-to-end driving model, comparing reconstruction quality, image-generation quality, and closed-loop behavior against both a simulation-only baseline and the real-world target. Results show that Gaussian-Splatting-based rendering reduces the visual gap with respect to standard simulator rendering and improves behavioral similarity to the real-world reference runs. The artifact is publicly available at https: //github.com/ast-fortiss-tum/cam2sim, and a screencast showing the tool is available at https://youtu.be/KmZ74l1__lI

Problem

• autonomous driving simulation
• sim-to-real gap
• camera observations
• scenario reconstruction
• closed-loop evaluation

Innovation

• Neural Scene Reconstruction
• Gaussian Splatting
• Autonomous Driving Simulation
• Closed-Loop Testing
• Camera-to-Simulation