DriveE2E: Closed-Loop Benchmark for End-to-End Autonomous Driving through Real-to-Simulation

2025-09-28

AI Summary

Existing CARLA-based closed-loop evaluation benchmarks rely on manually designed scenarios, suffering from limited realism and poor correlation with real-world driving performance. To address this, we propose the first high-fidelity closed-loop evaluation framework grounded in real-world roadside perception data. Leveraging 100 hours of real driving footage, we extract 800 dynamic traffic scenarios and reconstruct digital twins of 15 complex urban intersections in CARLA. Our method innovatively integrates 3D static scene reconstruction, vision-based alignment, and infrastructure-guided trajectory generation to ensure both visual and behavioral fidelity in simulation-to-reality transfer. The resulting benchmark spans diverse weather conditions, times of day, and geographic settings, substantially enhancing scenario realism and evaluation rigor. We open-source a comprehensive closed-loop test suite, enabling more realistic assessment of end-to-end autonomous driving models and advancing simulation-based testing from “human-designed” toward “reality-driven” paradigms.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Computer Vision: Diffusion Models for Vision

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

Closed-loop evaluation is increasingly critical for end-to-end autonomous driving. Current closed-loop benchmarks using the CARLA simulator rely on manually configured traffic scenarios, which can diverge from real-world conditions, limiting their ability to reflect actual driving performance. To address these limitations, we introduce a simple yet challenging closed-loop evaluation framework that closely integrates real-world driving scenarios into the CARLA simulator with infrastructure cooperation. Our approach involves extracting 800 dynamic traffic scenarios selected from a comprehensive 100-hour video dataset captured by high-mounted infrastructure sensors, and creating static digital twin assets for 15 real-world intersections with consistent visual appearance. These digital twins accurately replicate the traffic and environmental characteristics of their real-world counterparts, enabling more realistic simulations in CARLA. This evaluation is challenging due to the diversity of driving behaviors, locations, weather conditions, and times of day at complex urban intersections. In addition, we provide a comprehensive closed-loop benchmark for evaluating end-to-end autonomous driving models. Project URL: href{https://github.com/AIR-THU/DriveE2E}{https://github.com/AIR-THU/DriveE2E}.

Problem

• Addresses limitations of manually configured traffic scenarios in CARLA simulator
• Integrates real-world driving scenarios into simulation with infrastructure cooperation
• Evaluates autonomous driving models using realistic digital twin intersections

Innovation

• Integrates real-world scenarios into CARLA simulator
• Extracts dynamic traffic from infrastructure sensor videos
• Creates digital twin assets for realistic intersection replication