🤖 AI Summary
To address the neglect of human visual attention in first-person video object detection, this paper proposes a gaze-guided Vision Transformer framework. It models eye-tracking trajectories as dynamic attention priors and injects them into the multi-head self-attention mechanism; a gaze-aware attention head importance metric is further introduced to explicitly modulate each head’s response strength to fixated regions. Additionally, depth information is fused to enhance spatial perception. The method achieves significant improvements over non-gaze-aware baselines—up to +3.2–5.7% mAP—on a custom simulator dataset and public benchmarks including Ego4D Ego-Motion and Ego-CH-Gaze. Ablation studies validate the effectiveness of both the gaze-guidance mechanism and the depth-fusion module. This work is the first to systematically demonstrate the interpretability value of eye movement signals for evaluating first-person object detection, establishing a novel paradigm for attention modeling in embodied intelligence.
📝 Abstract
Human gaze offers rich supervisory signals for understanding visual attention in complex visual environments. In this paper, we propose Eyes on Target, a novel depth-aware and gaze-guided object detection framework designed for egocentric videos. Our approach injects gaze-derived features into the attention mechanism of a Vision Transformer (ViT), effectively biasing spatial feature selection toward human-attended regions. Unlike traditional object detectors that treat all regions equally, our method emphasises viewer-prioritised areas to enhance object detection. We validate our method on an egocentric simulator dataset where human visual attention is critical for task assessment, illustrating its potential in evaluating human performance in simulation scenarios. We evaluate the effectiveness of our gaze-integrated model through extensive experiments and ablation studies, demonstrating consistent gains in detection accuracy over gaze-agnostic baselines on both the custom simulator dataset and public benchmarks, including Ego4D Ego-Motion and Ego-CH-Gaze datasets. To interpret model behaviour, we also introduce a gaze-aware attention head importance metric, revealing how gaze cues modulate transformer attention dynamics.