TextGaze: Prompting Gaze Target Estimation with Textual Scene Cues

2026-07-11

AI Summary

This work addresses the limitations of existing gaze target estimation methods, which either suffer from high annotation costs and poor generalization due to multi-branch architectures or exhibit prediction bias by relying solely on low-level saliency while neglecting true gaze intent. To overcome these issues, we propose TextGaze, the first approach to leverage a frozen large vision-language model (LVLM) as a scalable source of semantic guidance. TextGaze extracts gaze-aligned textual cues from the LVLM and integrates them via a Transformer-based fusion module with hierarchical text supervision, followed by a lightweight decoder that jointly predicts the gaze heatmap and whether the gaze lies within the image frame. Without fine-tuning the LVLM, our method unifies multi-branch semantic modeling with an efficient architecture, achieving state-of-the-art performance across four benchmark datasets and demonstrating strong cross-dataset generalization.

Abstract

Gaze target estimation aims to infer the position of a person's gaze within a scene. Within mainstream design logic, multi-branch methods require extra supervision and annotations, while streamlined designs prioritize low-level visual saliency over true gaze intent. The former leads to a high annotation burden and hinders domain transfer, whereas the latter causes misalignment between predicted attention and actual gaze targets. To address this issue, we propose TextGaze, a unified cross-modal architecture that leverages a Large Vision-Language Model (LVLM) as scalable semantic guidance to balance the two design paradigms. The model extracts visual features from a frozen encoder and utilizes an LVLM to obtain gaze-aligned textual cues. We design a transformer-based fusion module with hierarchical text supervision to preserve task semantics. Lightweight decoding heads enable the joint prediction of gaze heatmaps and in-/out-of-frame status. We evaluate our method on four mainstream datasets, and the results show competitive performance across key metrics with robust cross-dataset generalisation without extra fine-tuning. Overall, we provide a streamlined alternative to traditional designs and highlight the potential of LVLMs as accessible auxiliary guidance for gaze estimation.

Problem

• gaze target estimation
• annotation burden
• visual saliency
• cross-domain generalization
• gaze intent

Innovation

• gaze target estimation
• vision-language model
• cross-modal fusion
• textual scene cues
• hierarchical supervision