S-NeRF++: Autonomous Driving Simulation via Neural Reconstruction and Generation

2024-02-03

arXiv.org

AI Summary

Existing autonomous driving simulators struggle to generate large-scale, photorealistic traffic data with complex scene dynamics, limiting model generalization. This paper introduces the first neural radiance field (NeRF)-based simulation system tailored for autonomous driving, enabling large-scale, high-fidelity reconstruction and synthesis of street scenes and dynamic vehicles using nuScenes and Waymo datasets. Our method innovatively integrates LiDAR-noise-robust optimization, learnable camera pose and scene parameterization, and physics-inspired illumination-consistent foreground-background compositing. These components jointly support rare-scenario editing, realistic shadow rendering, and diverse vehicle asset generation. Evaluated on object detection and bird’s-eye-view (BEV) segmentation, our synthetic data improves mAP by 2.1–3.8%. Moreover, the system enables efficient construction of over one thousand high-quality, editable simulation scenes—marking a significant advance in scalable, realistic, and controllable autonomous driving simulation.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Computer Vision: Vision for Robotics & Autonomous Driving
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

Autonomous driving simulation system plays a crucial role in enhancing self-driving data and simulating complex and rare traffic scenarios, ensuring navigation safety. However, traditional simulation systems, which often heavily rely on manual modeling and 2D image editing, struggled with scaling to extensive scenes and generating realistic simulation data. In this study, we present S-NeRF++, an innovative autonomous driving simulation system based on neural reconstruction. Trained on widely-used self-driving datasets such as nuScenes and Waymo, S-NeRF++ can generate a large number of realistic street scenes and foreground objects with high rendering quality as well as offering considerable flexibility in manipulation and simulation. Specifically, S-NeRF++ is an enhanced neural radiance field for synthesizing large-scale scenes and moving vehicles, with improved scene parameterization and camera pose learning. The system effectively utilizes noisy and sparse LiDAR data to refine training and address depth outliers, ensuring high-quality reconstruction and novel-view rendering. It also provides a diverse foreground asset bank by reconstructing and generating different foreground vehicles to support comprehensive scenario creation.Moreover, we have developed an advanced foreground-background fusion pipeline that skillfully integrates illumination and shadow effects, further enhancing the realism of our simulations. With the high-quality simulated data provided by our S-NeRF++, we found the perception methods enjoy performance boosts on several autonomous driving downstream tasks, further demonstrating our proposed simulator's effectiveness.

Problem

• Autonomous Driving Simulation
• Large-scale Scenarios
• Realism Limitations

Innovation

• Neural Representation
• Sparse LiDAR Optimization
• Foreground-Background Combination