Unraveling the Effects of Synthetic Data on End-to-End Autonomous Driving

2025-03-23

AI Summary

Current autonomous driving simulators face three key bottlenecks: game engines struggle to generate photorealistic sensor data; neural radiance fields (NeRF) and diffusion models suffer from low inference efficiency; and most closed-loop simulators lack expressive multi-vehicle interaction modeling—collectively limiting data diversity and generalization of end-to-end models. To address these, we propose SceneCrafter—the first high-fidelity, interactive autonomous driving simulator built upon 3D Gaussian Splatting (3DGS). SceneCrafter uniquely integrates 3DGS-based photorealistic rendering, closed-loop driving simulation, and multi-agent traffic interaction modeling, achieving a balanced trade-off among fidelity, dynamic interactivity, and generation efficiency. Experiments demonstrate that end-to-end models trained on SceneCrafter-synthesized data exhibit significantly improved generalization to unseen scenarios, with an average cross-scenario performance gain of 21.4%. Moreover, SceneCrafter provides a robust platform for reliable closed-loop evaluation.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

End-to-end (E2E) autonomous driving (AD) models require diverse, high-quality data to perform well across various driving scenarios. However, collecting large-scale real-world data is expensive and time-consuming, making high-fidelity synthetic data essential for enhancing data diversity and model robustness. Existing driving simulators for synthetic data generation have significant limitations: game-engine-based simulators struggle to produce realistic sensor data, while NeRF-based and diffusion-based methods face efficiency challenges. Additionally, recent simulators designed for closed-loop evaluation provide limited interaction with other vehicles, failing to simulate complex real-world traffic dynamics. To address these issues, we introduce SceneCrafter, a realistic, interactive, and efficient AD simulator based on 3D Gaussian Splatting (3DGS). SceneCrafter not only efficiently generates realistic driving logs across diverse traffic scenarios but also enables robust closed-loop evaluation of end-to-end models. Experimental results demonstrate that SceneCrafter serves as both a reliable evaluation platform and a efficient data generator that significantly improves end-to-end model generalization.

Problem

• Synthetic data limitations in autonomous driving models
• Lack of realistic and interactive traffic simulation
• Need for efficient and diverse data generation

Innovation

• Uses 3D Gaussian Splatting for realistic simulation
• Generates diverse traffic scenarios efficiently
• Enables robust closed-loop model evaluation