From Scene to Object: Text-Guided Dual-Gaze Prediction

2026-04-22

AI Summary

Existing driver attention datasets provide only scene-level gaze heatmaps and lack object-level annotations, limiting the interpretability of vision-language models and often leading to visual bias and hallucination. To address this, this work introduces G-W3DA, the first object-level driver attention dataset, and proposes DualGaze-VLM, a dual-branch architecture that integrates a multimodal large language model with SAM3 to decompose scene-level heatmaps into object masks. By leveraging semantic query hidden states and a condition-aware SE gating mechanism, the model enables text-guided, precise object-level gaze prediction. Evaluated on the W3DA benchmark, the method significantly outperforms state-of-the-art approaches, achieving a 17.8% improvement in the SIM metric for safety-critical scenarios, with 88.22% of human evaluators rating its generated heatmaps as realistic and trustworthy.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Security and Privacy: Large-scale security measurements
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Interpretable driver attention prediction is crucial for human-like autonomous driving. However, existing datasets provide only scene-level global gaze rather than fine-grained object-level annotations, inherently failing to support text-grounded cognitive modeling. Consequently, while Vision-Language Models (VLMs) hold great potential for semantic reasoning, this critical data limitations leads to severe text-vision decoupling and visual-bias hallucinations. To break this bottleneck and achieve precise object-level attention prediction, this paper proposes a novel dual-branch gaze prediction framework, establishing a complete paradigm from data construction to model architecture. First, we construct G-W3DA, a object-level driver attention dataset. By integrating a multimodal large language model with the Segment Anything Model 3 (SAM3), we decouple macroscopic heatmaps into object-level masks under rigorous cross-validation, fundamentally eliminating annotation hallucinations. Building upon this high-quality data foundation, we propose the DualGaze-VLM architecture. This architecture extracts the hidden states of semantic queries and dynamically modulates visual features via a Condition-Aware SE-Gate, achieving intent-driven precise spatial anchoring. Extensive experiments on the W3DA benchmark demonstrate that DualGaze-VLM consistently surpasses existing state-of-the-art (SOTA) models in spatial alignment metrics, notably achieving up to a 17.8% improvement in Similarity (SIM) under safety-critical scenarios. Furthermore, a visual Turing test reveals that the attention heatmaps generated by DualGaze-VLM are perceived as authentic by 88.22% of human evaluators, proving its capability to generate rational cognitive priors.

Problem

• driver attention prediction
• object-level annotation
• text-vision decoupling
• visual-bias hallucinations
• Vision-Language Models

Innovation

• object-level gaze prediction
• Vision-Language Models
• multimodal grounding
• attention hallucination mitigation
• Condition-Aware SE-Gate