System-Mediated Attention Imbalances Make Vision-Language Models Say Yes

2026-01-18

AI Summary

Vision-language models often exhibit a “yes”-bias hallucination due to imbalanced attention allocation among the system, image, and text modalities. This work identifies redundant attention in the system modality as a key contributor to this bias—a factor overlooked by prior studies that focused solely on image-side attention. To address this, the authors propose a causal attention reallocation method that dynamically rebalances weights across the three modalities to strengthen reliance on both visual and textual inputs. Experimental results demonstrate that the proposed approach significantly mitigates “yes”-bias hallucination, outperforming existing methods across multiple benchmarks. Furthermore, the study reveals that coarse-grained representation dependence constitutes a critical mechanism underlying this bias.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Vision-language model (VLM) hallucination is commonly linked to imbalanced allocation of attention across input modalities: system, image and text. However, existing mitigation strategies tend towards an image-centric interpretation of these imbalances, often prioritising increased image attention while giving less consideration to the roles of the other modalities. In this study, we evaluate a more holistic, system-mediated account, which attributes these imbalances to functionally redundant system weights that reduce attention to image and textual inputs. We show that this framework offers a useful empirical perspective on the yes-bias, a common form of hallucination in which VLMs indiscriminately respond'yes'. Causally redistributing attention from the system modality to image and textual inputs substantially suppresses this bias, often outperforming existing approaches. We further present evidence suggesting that system-mediated attention imbalances contribute to the yes-bias by encouraging a default reliance on coarse input representations, which are effective for some tasks but ill-suited to others. Taken together, these findings firmly establish system attention as a key factor in VLM hallucination and highlight its potential as a lever for mitigation.

Problem

• vision-language models
• hallucination
• yes-bias
• attention imbalance
• system modality

Innovation

• system-mediated attention
• vision-language models
• hallucination
• yes-bias
• attention redistribution