Visual Access Boundaries in Vision-Language Model Reasoning

2026-07-14

AI Summary

This study investigates whether vision-language models continuously rely on image tokens during chain-of-thought (CoT) reasoning or merely leverage early visual information. To address this, we introduce the concept of a “visual access boundary” and develop a corresponding visual access scanning intervention framework, integrating causal interventions, attention masking, temporal and layer-depth controls, symbolic attribute oracles, and single-object probe decoding to systematically analyze the model’s actual dependence on image tokens. Experiments across multi-scale architectures, including Qwen2.5-VL and InternVL3, reveal that CoT performance gains primarily stem from expanded language-side computation rather than sustained image access. Furthermore, the capacity to reliably read out perceptual visual attributes emerges as the key bottleneck limiting reasoning improvements.

Abstract

Chain-of-Thought (CoT) prompting is widely used as a test-time scaling strategy for Vision-Language Models (VLMs), but it remains unclear what is extended when VLMs generate longer reasoning traces. We ask whether CoT requires continued access to image tokens, or whether it mainly operates over visual information already made available earlier in the forward pass. We introduce Visual Access Sweep, a causal intervention that masks attention from generated-token queries to image-token keys along layer depth and generation time, and define the Visual Access Boundary (VAB) as the minimal access region that preserves task accuracy. Across six model configurations from Qwen2.5-VL and InternVL3, both no-CoT direct answering and CoT prompting exhibit finite VABs. In Qwen2.5-VL-32B and InternVL3 at 14B and 38B scales, when CoT is evaluated against the no-CoT full-access target, its VAB layer differs from the no-CoT boundary by at most two layers, despite substantially longer generations. This suggests that CoT does not primarily improve performance by prolonging direct image-token access throughout the reasoning trace, but by extending language-side computation over image-derived hidden-state information. We further show that CoT gains are constrained by perceptual readout. CoT helps when the queried visual attribute can be reliably read out by the model, but not when that readout is unreliable. A symbolic-attribute oracle shows that CoT can improve counting once ground-truth attributes are supplied as text, while a single-object probe-vs-decode check shows that hard attributes can be linearly recoverable from hidden states yet difficult for the model itself to output. Together, these analyses place the bottleneck at readout rather than counting.

Problem

• Vision-Language Models
• Chain-of-Thought
• Visual Access Boundary
• Perceptual Readout
• Reasoning

Innovation

• Visual Access Boundary
• Chain-of-Thought
• Vision-Language Models
• causal intervention
• perceptual readout