Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Survey

2025-03-17

AI Summary

This work addresses core challenges in learning-based 3D reconstruction for autonomous driving—namely, low dynamic modeling accuracy, insufficient multimodal fusion, and weak task adaptability. To this end, we propose the first multi-dimensional taxonomy specifically designed for autonomous driving, categorizing methods along three axes: input modalities, static/dynamic scene modeling paradigms, and task-oriented design principles. We further introduce the first hardware–sensor–aware method selection guide, systematically distilling technological evolution trends and critical bottlenecks. Our framework unifies NeRFs, TSDFs, point cloud networks, and multiview geometry to jointly leverage LiDAR, camera, and IMU data for high-fidelity reconstruction. Additionally, we establish a unified evaluation protocol covering data formats, subtask decomposition, benchmarking suites, and performance comparison. Experimental results demonstrate significant improvements in downstream tasks, including scene understanding and closed-loop simulation.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Learning-based 3D reconstruction has emerged as a transformative technique in autonomous driving, enabling precise modeling of both dynamic and static environments through advanced neural representations. Despite augmenting perception, 3D reconstruction inspires pioneering solution for vital tasks in the field of autonomous driving, such as scene understanding and closed-loop simulation. Commencing with an examination of input modalities, we investigates the details of 3D reconstruction and conducts a multi-perspective, in-depth analysis of recent advancements. Specifically, we first provide a systematic introduction of preliminaries, including data formats, benchmarks and technical preliminaries of learning-based 3D reconstruction, facilitating instant identification of suitable methods based on hardware configurations and sensor suites. Then, we systematically review learning-based 3D reconstruction methods in autonomous driving, categorizing approaches by subtasks and conducting multi-dimensional analysis and summary to establish a comprehensive technical reference. The development trends and existing challenges is summarized in the context of learning-based 3D reconstruction in autonomous driving. We hope that our review will inspire future researches.

Problem

• Enhances 3D reconstruction for autonomous driving environments.
• Explores advancements in scene understanding and simulation.
• Reviews and categorizes learning-based 3D reconstruction methods.

Innovation

• Advanced neural representations for 3D reconstruction
• Multi-perspective analysis of 3D reconstruction methods
• Systematic review of learning-based 3D reconstruction techniques