AUTOPILOT VQA: Benchmarking Vision-Language Models for Incident-Centric Dashcam Understanding

2026-07-09

AI Summary

Existing vision-language models lack systematic evaluation of their temporal reasoning capabilities in safety-critical driving scenarios. This work proposes the first dashcam video–centered visual question answering benchmark focused on real-world accidents and near-miss events. It constructs structured questions spanning multiple dimensions—including weather conditions, road states, traffic signs, and involved entities—to assess models’ ability to temporally ground scene context and event details. Built upon authentic driving videos and integrated with a multimodal large model evaluation framework, the benchmark supports the development of interpretable and robust vision-language systems for autonomous driving. It will serve as the official dataset for the CVPR 2026 competition, providing a unified platform for evaluating the reliability of autonomous driving systems across diverse safety-critical scenarios.

Abstract

Recent advances in Vision-Language Models, Large Language Models, and Multimodal Large Language Models have improved autonomous driving tasks such as scene understanding, decision making, trajectory prediction, and visual question answering. However, evaluating whether these models can reliably reason about safety-critical incidents remains challenging. To address this gap, we present AUTOPILOT-VQA, an incident-centric visual question answering benchmark for dashcam video understanding. The dataset evaluates different systems through structured questions designed around real-world driving incidents and near-incidents. The benchmark covers diverse safety-relevant categories, including weather and lighting conditions, traffic environment, road layout, road surface state, signage, involved entities, accident occurrence, impact location, and avoidability-related reasoning. By requiring models to answer grounded questions about both contextual scene properties and event-level incident details, AUTOPILOT-VQA moves beyond object recognition toward temporally grounded, safety-aware reasoning. The dataset is released as part of the AUTOPILOT CVPR 2026 competition and provides a standardized benchmark for assessing the reliability of autonomous driving systems in different scenarios. Our benchmark support developments for more interpretable, robust, and safety-conscious vision-language systems for real-world autonomous driving.

Problem

• visual question answering
• autonomous driving
• safety-critical incidents
• vision-language models
• dashcam understanding

Innovation

• incident-centric VQA
• dashcam video understanding
• safety-aware reasoning
• temporally grounded reasoning
• autonomous driving benchmark