Safety-Aware Evaluation of LLM-Generated Driver Intervention Messages through Multi-Task Risk Fusion

2026-06-21

AI Summary

Current driver intervention systems rely on fixed templates and generic metrics, struggling to accurately evaluate the quality of intervention messages along safety-critical dimensions such as risk urgency alignment, cognitive load, and acceptability. This work proposes an end-to-end intervention generation framework that integrates outputs from four recognition tasks, leveraging multi-task risk fusion, state history management, and dynamic prompt generation to drive a lightweight large language model (LLM) in producing context-aware interventions. Furthermore, it introduces DSAIS—the first multidimensional evaluation framework tailored for driving interventions. Experimental results demonstrate that DSAIS exhibits high reliability and validity across three rater groups (ICC = 0.798–0.840; Cohen’s d > 1.5), with locally deployed 7B–9B parameter LLMs outperforming API-based models. Emotion recognition emerges as the most critical upstream factor, and multi-task fusion significantly improves intervention relevance by 9.1% over rule-based systems.

Abstract

Existing driver intervention systems rely on auditory alerts and fixed templates, failing to leverage multi-task recognition outputs. General-purpose metrics such as BLEU and BERTScore cannot capture intervention-specific quality dimensions including risk-urgency alignment, cognitive load, and driver acceptability. In this paper, we propose the Driver Safety-Aware Intervention Score (DSAIS), a domain-specific metric evaluating five dimensions through a hybrid architecture combining lightweight rule-based computation with LLM Judge evaluation, together with an end-to-end framework integrating four-task recognition outputs into an LLM through risk fusion, state history management, and dynamic prompt construction. Experiments on the AIDE dataset with five models and seven conditions demonstrate that DSAIS achieves ICC 0.798-0.840 across three architecturally distinct judges and Cohen's d > 1.5 across all control conditions. Multi-dimensional sub-score analysis quantifies the contextual adaptability gap between rule-based and LLM-based systems, revealing that multi-task integration improves contextual relevance by 9.1% over rule-based baselines. Ablation experiments demonstrate that each framework component contributes to contextual relevance, with sub-score decomposition revealing gains that aggregate scoring masks. Driver emotion recognition is identified as the most critical upstream factor, and compact local LLMs (7B--9B parameters) achieve quality superior to API-based models, providing practical design guidelines for in-vehicle deployment.

Problem

• driver intervention
• LLM evaluation
• safety-aware
• multi-task fusion
• risk-urgency alignment

Innovation

• risk fusion
• domain-specific evaluation
• multi-task integration
• dynamic prompt construction
• compact LLM deployment