Attention Alignment Between Humans and Vision-Language Models

2026-06-15

AI Summary

This study investigates the alignment between attention mechanisms in vision-language models (VLMs) and human visual attention at both behavioral and neural levels. By systematically comparing spatial attention maps from six VLM architectures—spanning combinations of CNN or ViT encoders with LSTM or Transformer decoders—against human eye-tracking fixation heatmaps, and evaluating their neural predictive power using TRIBE-synthesized neural responses, the work reveals that decoder architecture predominantly governs behavioral alignment (with LSTMs achieving 85–87% of the noise ceiling), whereas the encoder plays a more critical role in predicting early visual cortex activity. The findings further demonstrate a dissociation between behavioral alignment and neural predictivity, and ablation experiments—including simulations of hemispatial neglect—validate the biological plausibility of the models’ attention mechanisms.

Abstract

Visual perception depends on top-down goals and bottom-up sensory mechanisms. Vision-language models implement both, allowing us to treat each component as a separable hypothesis about what drives where we look. We compared spatial attention maps from six vision-language models against human fixation heatmaps recorded on 200 images during two tasks (general description and social captioning). The six models spanned a 2$\times$2 factorial of CNN vs.\ ViT encoders crossed with LSTM vs.\ Transformer decoders, plus Molmo 7B-D and Qwen3.5 9B. We found that both decoder and encoder architecture shaped alignment, but decoder choice dominated. LSTM vs.\ Transformer decoders increased alignment by 40--50 percentage points (80--87\% vs.\ 40--59\% of the human noise ceiling). In contrast, CNN vs.\ ViT encoders contributed a secondary 5--20 point advantage depending on decoder family, with CNN-LSTM the most aligned model overall (85--87\%). Despite their alignment advantage, LSTM-decoder attention maps were spatially diffuse and minimally task-differentiated; ViT-Transformer, the weakest in alignment, showed the sharpest spatial concentration and strongest task differentiation. A hemispatial-neglect simulation confirmed that ablating attention impacted LSTM decoders more than Transformer decoders. In an exploratory extension using TRIBE-simulated synthetic neural responses, fixation alignment and neural relevance dissociate: CNN-Transformer attention maps better predicted synthetic brain activity despite lower fixation alignment, with attention maps best predicting early visual cortex. Together, top-down and bottom-up components trade off what they predict in behavioral and synthetic neural data.

Problem

• attention alignment
• vision-language models
• human fixation
• model architecture
• neural relevance

Innovation

• attention alignment
• vision-language models
• human fixation
• neural relevance
• encoder-decoder architecture