Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts

2026-04-10

AI Summary

This study investigates the root causes of errors in vision-language models when visual and linguistic priors conflict, distinguishing between perceptual deficits and failures in multimodal arbitration. Through Multimodal Arbitration Cross (MAC) analysis, layer-wise Logit Lens probing, and full-sequence activation patching, the authors find that early model layers already encode visual attributes with high fidelity (AUC > 0.86), yet final predictions disregard visual evidence due to faulty arbitration. The work identifies a “encoding-grounding decoupling” phenomenon and demonstrates that image tokens carry the dominant causal effect. Furthermore, it introduces a training-free activation steering method that improves visual grounding by up to 3.8% and shows that activation patching alters 60–84% of model outputs, confirming that the core issue lies in arbitration rather than perception.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

When a Vision-Language Model (VLM) sees a blue banana and answers"yellow", is the problem of perception or arbitration? We explore the question in ten VLMs with various sizes and reveal an Encoding--Grounding Dissociation: models that fail to report what they see (and thus provide a wrong answer) still encode the visual evidence as strongly as models that provide the correct answer. Using Multimodal Arbitration Crossover (MAC) analysis with layer-by-layer Logit Lens probing, we track the competition between visual and prior signals across every layer of each model. We show that visual attributes can be linearly decodable from early layers (AUC>0.86). The accuracy remains nearly identical for both successful and failed samples. However, the gap in the final-layer logit -- not the strength of encoding -- better predicts grounding outcomes with a correlation of . After having studied when VLMs base their answers on image clues rather than prior knowledge, we want to understand the causal relationships. We establish causality through full-sequence activation patching. The standard last-token interventions in LLM interpretability do not affect VLMs. In contrast, replacing the full token sequence at layers identified by MAC alters 60 to 84% of outputs. Partial-token decomposition shows that image tokens carry almost all of the causal impact, while text tokens have none. Scaling addresses the remaining architectural differences to achieve perfect retention. Moving from diagnosis to intervention, we show that training-free activation steering -- both linear and sparse autoencoder-guided -- in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups. Overall, these findings lead to a clear conclusion: VLMs already see well, but the challenge is acting on what they see. Targeted interventions can help to bridge this gap.

Problem

• Vision-Language Models
• Visual-Linguistic Conflicts
• Perceptual Blindness
• Multimodal Arbitration
• Visual Grounding

Innovation

• Multimodal Arbitration Crossover
• Logit Lens probing
• activation patching
• visual grounding
• activation steering