HetroD: A High-Fidelity Drone Dataset and Benchmark for Autonomous Driving in Heterogeneous Traffic

2026-02-03

AI Summary

Existing autonomous driving datasets struggle to capture the complex, unstructured interactions of vulnerable road users—such as pedestrians and cyclists—in high-density heterogeneous traffic, including behaviors like hook turns and lane weaving. To address this gap, this work presents a large-scale, high-fidelity traffic dataset collected via drone-based aerial imagery, featuring centimeter-accurate trajectory annotations, high-definition maps, and synchronized traffic signal states. A modular toolchain is provided to support downstream tasks. The released dataset comprises over 65.4k trajectories, with 70% corresponding to vulnerable road users, and establishes the first standardized benchmark for such scenarios. Evaluations reveal a significant performance drop in current prediction and planning models, underscoring the need for novel approaches tailored to these challenging interactions.

Technology Category

• Application Domains: Mobility, Driving & Flight
• Computer Vision: Vision for Robotics & Autonomous Driving
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Security and Privacy: Data transparency and provenance

Abstract

We present HetroD, a dataset and benchmark for developing autonomous driving systems in heterogeneous environments. HetroD targets the critical challenge of navi- gating real-world heterogeneous traffic dominated by vulner- able road users (VRUs), including pedestrians, cyclists, and motorcyclists that interact with vehicles. These mixed agent types exhibit complex behaviors such as hook turns, lane splitting, and informal right-of-way negotiation. Such behaviors pose significant challenges for autonomous vehicles but remain underrepresented in existing datasets focused on structured, lane-disciplined traffic. To bridge the gap, we collect a large- scale drone-based dataset to provide a holistic observation of traffic scenes with centimeter-accurate annotations, HD maps, and traffic signal states. We further develop a modular toolkit for extracting per-agent scenarios to support downstream task development. In total, the dataset comprises over 65.4k high- fidelity agent trajectories, 70% of which are from VRUs. HetroD supports modeling of VRU behaviors in dense, het- erogeneous traffic and provides standardized benchmarks for forecasting, planning, and simulation tasks. Evaluation results reveal that state-of-the-art prediction and planning models struggle with the challenges presented by our dataset: they fail to predict lateral VRU movements, cannot handle unstructured maneuvers, and exhibit limited performance in dense and multi-agent scenarios, highlighting the need for more robust approaches to heterogeneous traffic. See our project page for more examples: https://hetroddata.github.io/HetroD/

Problem

• heterogeneous traffic
• vulnerable road users
• autonomous driving
• unstructured behavior
• trajectory prediction

Innovation

• heterogeneous traffic
• vulnerable road users (VRUs)
• drone-based dataset
• high-fidelity trajectory
• autonomous driving benchmark