Pathways of Visual Information Flow in Vision-Language Models

2026-07-03

AI Summary

This study investigates the pathways and dependency mechanisms through which visual information is processed in vision-language models. By employing causal mediation analysis, attention ablation, and corrupted input repair as intervention techniques, the work systematically uncovers two switchable visual processing routes: a direct visual pathway and a text-mediated pathway. The choice between these pathways is shown to be significantly influenced by task type, data distribution, and prompt design. Notably, when the primary pathway is obstructed, the model flexibly activates the alternative route to preserve performance. This framework reconciles previously conflicting observations regarding model behavior and demonstrates the controllability and robustness of these pathways across three distinct vision tasks.

Abstract

We study how visual information is routed in vision-language models (VLMs). Using causal patching on controlled synthetic and natural datasets, we find that models rely on two distinct pathways to solve visual tasks: A direct pathway, where visual information is retained in image token representations and read out by the final token at later layers, and a text-mediated pathway, where visual information is first transferred to the query tokens and then read out by the final token. Across three visual tasks, we show that pathway selection is task-dependent, and that data distribution and prompt design can also modulate which pathway is used to solve the image-based query. Moreover, using attention knockouts and corrupted-input patching, we find that these pathways are flexible, under certain interventions, models can rely on the text-mediated pathway as a fallback when the usual pathway is ablated. This behavior unifies findings in prior work and shows that ablation-based interventions can reveal what models could do rather than what they normally do. Together, our results provide a mechanistic characterization of visual information flow in VLMs and highlight the flexibility of their internal mechanisms under intervention.

Problem

• vision-language models
• visual information flow
• information pathways
• causal patching
• model interpretability

Innovation

• vision-language models
• information pathways
• causal patching
• attention knockouts
• mechanistic interpretability