NuRisk: A Visual Question Answering Dataset for Agent-Level Risk Assessment in Autonomous Driving

2025-09-30

AI Summary

Current vision-language models (VLMs) rely on static images and thus struggle to model the spatiotemporal dynamics of risk in autonomous driving, lacking quantitative risk assessment and reasoning capabilities. To address this, we introduce NuRisk—the first fine-grained visual question answering (VQA) dataset for agent-level risk assessment—comprising 2,900 real-world and simulated scenarios from nuScenes, Waymo, and CommonRoad, annotated with quantitative risk labels over bird’s-eye-view (BEV) temporal sequences to enable explicit spatiotemporal reasoning. Benchmarking reveals that state-of-the-art VLMs achieve only 33% accuracy in dynamic risk judgment with high latency; instruction-tuned 7B VLMs improve accuracy to 41% and reduce latency by 75%, validating the efficacy of spatiotemporal-aware risk modeling. Our core contribution is the first VQA benchmark supporting quantitative spatiotemporal reasoning for risk assessment, advancing VLMs toward safety-critical, dynamic decision-making.

Technology Category

• Computer Vision: Language and Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Safety and Robustness

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Understanding risk in autonomous driving requires not only perception and prediction, but also high-level reasoning about agent behavior and context. Current Vision Language Models (VLMs)-based methods primarily ground agents in static images and provide qualitative judgments, lacking the spatio-temporal reasoning needed to capture how risks evolve over time. To address this gap, we propose NuRisk, a comprehensive Visual Question Answering (VQA) dataset comprising 2,900 scenarios and 1.1 million agent-level samples, built on real-world data from nuScenes and Waymo, supplemented with safety-critical scenarios from the CommonRoad simulator. The dataset provides Bird-Eye-View (BEV) based sequential images with quantitative, agent-level risk annotations, enabling spatio-temporal reasoning. We benchmark well-known VLMs across different prompting techniques and find that they fail to perform explicit spatio-temporal reasoning, resulting in a peak accuracy of 33% at high latency. To address these shortcomings, our fine-tuned 7B VLM agent improves accuracy to 41% and reduces latency by 75%, demonstrating explicit spatio-temporal reasoning capabilities that proprietary models lacked. While this represents a significant step forward, the modest accuracy underscores the profound challenge of the task, establishing NuRisk as a critical benchmark for advancing spatio-temporal reasoning in autonomous driving.

Problem

• Addresses autonomous driving risk assessment requiring spatio-temporal reasoning
• Proposes VQA dataset with agent-level risk annotations for dynamic scenarios
• Benchmarks VLMs' limitations in explicit spatio-temporal reasoning for risk evolution

Innovation

• Developed NuRisk VQA dataset with sequential BEV images
• Fine-tuned 7B VLM for explicit spatio-temporal reasoning
• Integrated real-world and simulated safety-critical scenarios