nuCarla: A nuScenes-Style Bird's-Eye View Perception Dataset for CARLA Simulation

2025-11-12

AI Summary

The lack of standardized, large-scale, and rigorously validated bird’s-eye view (BEV) perception datasets hinders the development of closed-loop end-to-end autonomous driving models. To address this, we introduce nuCarla—a large-scale, high-fidelity synthetic BEV dataset built on CARLA and fully compliant with the nuScenes data format. It provides multi-view sensor data (cameras, LiDAR), precise 3D annotations, and structured BEV representations. Our key contribution is the first end-to-end adaptation of the nuScenes specification within CARLA, enabling balanced class distributions and high-quality, ready-to-use open- and closed-loop data. We further propose a high-performance BEV backbone network that achieves state-of-the-art results on BEV object detection benchmarks. nuCarla significantly accelerates training, evaluation, and iterative validation of end-to-end autonomous driving systems.

Technology Category

• Computer Vision: Vision for Robotics & Autonomous Driving
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Application Domains: Mobility, Driving & Flight

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Sustainability and carbon-aware systems for Web, mobile, and WoT

Abstract

End-to-end (E2E) autonomous driving heavily relies on closed-loop simulation, where perception, planning, and control are jointly trained and evaluated in interactive environments. Yet, most existing datasets are collected from the real world under non-interactive conditions, primarily supporting open-loop learning while offering limited value for closed-loop testing. Due to the lack of standardized, large-scale, and thoroughly verified datasets to facilitate learning of meaningful intermediate representations, such as bird's-eye-view (BEV) features, closed-loop E2E models remain far behind even simple rule-based baselines. To address this challenge, we introduce nuCarla, a large-scale, nuScenes-style BEV perception dataset built within the CARLA simulator. nuCarla features (1) full compatibility with the nuScenes format, enabling seamless transfer of real-world perception models; (2) a dataset scale comparable to nuScenes, but with more balanced class distributions; (3) direct usability for closed-loop simulation deployment; and (4) high-performance BEV backbones that achieve state-of-the-art detection results. By providing both data and models as open benchmarks, nuCarla substantially accelerates closed-loop E2E development, paving the way toward reliable and safety-aware research in autonomous driving.

Problem

• Lack of standardized datasets for closed-loop autonomous driving simulation
• Limited availability of bird's-eye-view perception data for interactive environments
• Insufficient resources for end-to-end autonomous driving model development

Innovation

• nuCarla dataset uses CARLA simulator for BEV perception
• It provides nuScenes-compatible format for model transfer
• Enables closed-loop simulation with high-performance BEV backbones