DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction

2026-07-02

AI Summary

This work addresses the limitations of traditional scanpath modeling, which relies on handcrafted architectures and struggles to flexibly incorporate conditional information such as task instructions or individual differences. For the first time, the authors introduce a vision-language model to this task by discretizing gaze coordinates into textual tokens and formulating scanpath prediction as an autoregressive sequence generation problem. Through prompt engineering, the framework jointly models multiple factors—including viewer identity, task goals, and fixation durations—within a unified architecture. The approach not only enables precise computation of information gain per fixation but also offers potential for behavioral intervention. Evaluated on MIT1003, it achieves an information gain of 2.18 bits, a 46% improvement over DeepGaze III, establishes new state-of-the-art results across multiple benchmarks, and successfully reproduces known eye movement phenomena.

Abstract

Understanding human visual attention on a scene over time has applications in domains such as interface design and inferring cognitive states. Modeling visual scanpaths has historically relied on specialized architectures with hand-crafted priors. While these architectures can model fixation sequences, their rigid structural biases restrict easy extendability and flexible conditioning. For instance, integrating task-specific instructions or adapting to distinct viewer identities requires custom, disjoint architectural additions. We frame scanpath prediction purely as a discrete sequence modeling task. By mapping coordinates into a text vocabulary, we leverage the pretrained representations of Vision-Language Models. This framing absorbs diverse factors of variation: simple prompting allows for global conditioning, such as providing viewer identities to capture personalized biases, or task-specific objectives like visual search. The framework can also integrate per-fixation attributes, such as individual fixation durations, alongside spatial locations. The autoregressive alignment enables the scalable, exact computation of per-fixation log-likelihoods, directly equivalent to the commonly used Information Gain (IG) metric. Our model, DeepGaze3.5-VL, establishes a new state-of-the-art across multiple datasets, achieving 2.18 bits of IG on MIT1003, a 46% improvement over DeepGaze III. This advantage persists even when baselines use identical high-capacity vision encoders. Beyond predictive performance, our generative framework serves as a powerful computational tool for direct behavioral interventions, allowing for controlled in-silico simulations that would be experimentally difficult or impossible to conduct in vivo. We demonstrate this ability by performing controlled interventions on the durations of pre-saccadic fixations, recovering known oculomotor phenomena purely from data.

Problem

• scanpath prediction
• visual attention
• sequence modeling
• vision-language models
• oculomotor behavior

Innovation

• scanpath modeling
• vision-language models
• autoregressive sequence modeling
• visual attention prediction
• computational behavioral intervention