nuScenes Revisited: Progress and Challenges in Autonomous Driving

2025-12-02

AI Summary

This paper systematically evaluates the impact of the nuScenes dataset on autonomous driving research. To address challenges in multimodal perception, localization and mapping, behavior prediction, and motion planning, the authors comprehensively document nuScenes’ construction methodology—including its extensions nuImages and Panoptic nuScenes—and highlight its pioneering design features: 4D radar modeling, cross-continental urban scene coverage, and full-stack autonomous vehicle data acquisition. Leveraging large-scale, real-world road data—fused from LiDAR, millimeter-wave radar, and multi-view cameras—with fine-grained annotations, nuScenes enables advances in 3D object detection, multi-object tracking, panoptic segmentation, and trajectory forecasting. The dataset has become a de facto benchmark in the field, significantly accelerating algorithmic progress and profoundly shaping subsequent dataset design principles and standardized evaluation protocols across the autonomous driving community.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Vision for Robotics & Autonomous Driving
• Machine Learning: Multimodal Learning

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Data transparency and provenance

Abstract

Autonomous Vehicles (AV) and Advanced Driver Assistance Systems (ADAS) have been revolutionized by Deep Learning. As a data-driven approach, Deep Learning relies on vast amounts of driving data, typically labeled in great detail. As a result, datasets, alongside hardware and algorithms, are foundational building blocks for the development of AVs. In this work we revisit one of the most widely used autonomous driving datasets: the nuScenes dataset. nuScenes exemplifies key trends in AV development, being the first dataset to include radar data, to feature diverse urban driving scenes from two continents, and to be collected using a fully autonomous vehicle operating on public roads, while also promoting multi-modal sensor fusion, standardized benchmarks, and a broad range of tasks including perception, localization &mapping, prediction and planning. We provide an unprecedented look into the creation of nuScenes, as well as its extensions nuImages and Panoptic nuScenes, summarizing many technical details that have hitherto not been revealed in academic publications. Furthermore, we trace how the influence of nuScenes impacted a large number of other datasets that were released later and how it defined numerous standards that are used by the community to this day. Finally, we present an overview of both official and unofficial tasks using the nuScenes dataset and review major methodological developments, thereby offering a comprehensive survey of the autonomous driving literature, with a particular focus on nuScenes.

Problem

• Revisiting nuScenes dataset creation and technical details
• Analyzing nuScenes' impact on later datasets and community standards
• Surveying autonomous driving tasks and methods using nuScenes

Innovation

• First dataset to include radar data
• Promotes multi-modal sensor fusion approach
• Establishes standardized benchmarks for tasks