NARRATE: A Multimodal Real-World Australian Driving Dataset for Human-Centred Explanations in Automated Driving

2026-08-14

AI Summary

This study addresses the scarcity of first-person linguistic annotations and situational awareness data in autonomous driving by constructing a multimodal real-world dataset comprising 35 drivers and 2,050 events. Uniquely, this dataset synchronizes sensor streams with drivers’ free-text explanations and fine-grained hierarchical situational labels, establishing four benchmark tasks. The research validates the feasibility of learning structured driving information from natural language while highlighting critical challenges in context recognition and explanation generation. By bridging this significant data gap, the work substantially advances human-centric explainability research in autonomous driving, providing essential resources for developing systems that better align with human cognitive processes and interpretability requirements.

Abstract

Automated vehicles must explain their decisions in ways that passengers can understand, monitor, and trust. Existing language-annotated driving datasets are mostly observer-written, post-hoc, simulation-based, or generated from sensor inputs, rather than elicited from the driver performing the action. We introduce NARRATE, a multimodal real-world Australian driving dataset comprising 2,050 annotated events from 35 experienced drivers and driving instructors on public roads. Each event is grounded in synchronised visual, localisation, motion, and LiDAR streams and paired with in-vehicle and/or post-drive free-text explanations. NARRATE provides action labels, scenario-context labels spanning six high-level and 32 fine-grained categories, and span-level Situational Awareness (SA) annotations over driver explanations for Perception, Comprehension and Projection. Four benchmark tasks (SA, scenario-context, driver-action classification, and explanation generation) show that this structure is learnable from driver language, while fine-grained context recognition and explanation generation remain challenging. NARRATE paves a path towards more human-centred and domain-aware explanation models for automated driving.

Problem

• Automated Driving
• Human-Centred Explanations
• Driving Dataset
• Situational Awareness

Innovation

• Multimodal Driving Dataset
• Human-Centred Explanations
• Situational Awareness Annotations
• Real-World Data
• Explanation Generation