Visionary Co-Driver: Enhancing Driver Perception of Potential Risks with LLM and HUD

2025-11-18

AI Summary

This study addresses drivers’ insufficient perception of non-collision road hazards—such as illegal parking, construction zones, and anomalous pedestrian behavior—by proposing a novel driver assistance system integrating large language models (LLMs), eye-tracking, and adaptive head-up display (HUD). Methodologically, it pioneers the use of LLMs for driving risk reasoning, combining video understanding algorithms to detect latent hazards and dynamically modulating HUD cue location, timing, and semantic intensity based on real-time gaze data—enabling context-aware, personalized hazard alerts. A user study (N=41) demonstrates significant improvements in both hazard identification accuracy and situation awareness for non-collision risks. Key contributions include: (1) establishing an LLM-driven paradigm for semantic risk inference; (2) closing the “vision–cognition–interaction” loop; and (3) providing a foundational framework for next-generation explainable, adaptive intelligent driver assistance systems.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Drivers' perception of risky situations has always been a challenge in driving. Existing risk-detection methods excel at identifying collisions but face challenges in assessing the behavior of road users in non-collision situations. This paper introduces Visionary Co-Driver, a system that leverages large language models to identify non-collision roadside risks and alert drivers based on their eye movements. Specifically, the system combines video processing algorithms and LLMs to identify potentially risky road users. These risks are dynamically indicated on an adaptive heads-up display interface to enhance drivers' attention. A user study with 41 drivers confirms that Visionary Co-Driver improves drivers' risk perception and supports their recognition of roadside risks.

Problem

• Enhancing driver perception of non-collision roadside risks
• Leveraging LLMs to identify potentially risky road user behavior
• Alerting drivers dynamically via adaptive heads-up display interface

Innovation

• Uses large language models for non-collision risk identification
• Combines video processing algorithms with LLMs
• Displays dynamic alerts on adaptive heads-up interface