Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models

2026-06-26

AI Summary

When visual evidence conflicts with world knowledge encoded in a model’s memory, the decision-making mechanism of vision-language models (VLMs) critically determines their reliability. This work systematically investigates the causal structure underlying such conflicts across three VLM families through activation patching, component ablation, and cross-granularity mechanistic analysis. The study identifies, for the first time at the component level, a sparse yet causally pivotal set of attention heads that implement an asymmetric “visual-default, prior-overrides” mechanism. Ablating these key heads reverses model predictions from knowledge-driven to vision-driven in 68%–96% of conflict cases, demonstrating their decisive role. Notably, this causal circuit exhibits remarkable consistency across different models, highlighting a shared mechanistic principle governing how VLMs resolve perceptual-knowledge discrepancies.

Abstract

Vision-language models must reconcile visual evidence with memorized world knowledge when the two conflict. How they resolve this conflict shapes the reliability of multimodal systems, yet prior work characterizes it behaviorally without a component-level causal account. We combine activation patching across three granularities (residual stream, attention heads, and MLP sublayers) with model-component ablation studies and mechanistic analysis. Across three VLM families, we find that visual grounding emerges by default, whereas prior grounding depends on a small set of causally necessary attention heads (2.5-4.8%) concentrated in the second half of the network. These heads enable answers from stored world knowledge (e.g., "red" for a strawberry) despite conflicting visual input. Ablating them flips predictions from knowledge-grounded to visually grounded answers in 68-96% of cases under prior-knowledge prompts, but changes only 0.8-7.5% of visually grounded predictions, establishing an asymmetric causal structure. The identified heads decompose into routing heads, which modulate information flow, and writing heads, which directly project answer tokens into the residual stream. This structure is consistent across model families and scales, revealing a sparse causal circuit underlying perception-knowledge conflict in VLMs.

Problem

• vision-language models
• perception-knowledge conflict
• visual grounding
• prior knowledge
• causal mechanisms

Innovation

• causal mechanism
• vision-language models
• activation patching
• attention heads
• perception-knowledge conflict