Toward Automatic Safe Driving Instruction: A Large-Scale Vision Language Model Approach

2025-11-28

AI Summary

This study addresses the challenge of jointly modeling driver state and road scene for driving safety monitoring. We propose the first dual-perspective (driver + forward-road) video understanding framework designed specifically for safety-critical driving guidance. Methodologically, we construct a synchronized dual-view video dataset, design a spatiotemporally aligned multimodal fusion mechanism, and perform fine-tuning of a large vision-language model (LVLM) for fine-grained safety instruction generation. Our core contribution is the introduction of a novel synchronized dual-perspective joint reasoning paradigm, enabling end-to-end generation of safety-aware driving instructions from heterogeneous visual inputs. Experimental results demonstrate that the fine-tuned model significantly outperforms baselines in instruction accuracy and safety awareness—e.g., reliably detecting high-risk behaviors such as mobile phone usage. However, generalization to subtle driver actions and complex human–environment interactions remains limited and warrants further improvement.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Safety and Robustness
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Security and Privacy: Large-scale security measurements
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large-scale Vision Language Models (LVLMs) exhibit advanced capabilities in tasks that require visual information, including object detection. These capabilities have promising applications in various industrial domains, such as autonomous driving. For example, LVLMs can generate safety-oriented descriptions of videos captured by road-facing cameras. However, ensuring comprehensive safety requires monitoring driver-facing views as well to detect risky events, such as the use of mobiles while driving. Thus, the ability to process synchronized inputs is necessary from both driver-facing and road-facing cameras. In this study, we develop models and investigate the capabilities of LVLMs by constructing a dataset and evaluating their performance on this dataset. Our experimental results demonstrate that while pre-trained LVLMs have limited effectiveness, fine-tuned LVLMs can generate accurate and safety-aware driving instructions. Nonetheless, several challenges remain, particularly in detecting subtle or complex events in the video. Our findings and error analysis provide valuable insights that can contribute to the improvement of LVLM-based systems in this domain.

Problem

• Developing LVLMs to generate safety-aware instructions from synchronized vehicle cameras
• Enhancing detection of risky driving events like mobile phone usage
• Improving performance on subtle or complex event recognition in videos

Innovation

• Fine-tuned LVLMs generate safety-aware driving instructions
• Process synchronized inputs from driver and road-facing cameras
• Construct dataset to evaluate LVLM performance on driving safety