Words or Vision: Do Vision-Language Models Have Blind Faith in Text?

2025-03-04

AI Summary

This work identifies a pervasive “textual overreliance” bias in vision-language models (VLMs): when image and text inputs are semantically inconsistent, VLMs excessively prioritize textual cues, severely degrading visual task performance and introducing safety risks. We systematically establish—through textual perturbations, instruction variations, and cross-model comparisons across four core vision-centric tasks—that this bias affects all ten mainstream VLMs evaluated, and we attribute it fundamentally to modality distribution imbalance in training data. To mitigate it, we propose a text-augmented supervised fine-tuning method, theoretically modeling its mechanism for enhancing multimodal alignment. Experiments demonstrate that our approach improves model robustness against modality inconsistency by an average of 23.6%. This work provides both a reproducible technical pathway and theoretical grounding for reducing modality bias and strengthening consistent multimodal reasoning in VLMs.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Vision-Language Models (VLMs) excel in integrating visual and textual information for vision-centric tasks, but their handling of inconsistencies between modalities is underexplored. We investigate VLMs' modality preferences when faced with visual data and varied textual inputs in vision-centered settings. By introducing textual variations to four vision-centric tasks and evaluating ten Vision-Language Models (VLMs), we discover a emph{``blind faith in text''} phenomenon: VLMs disproportionately trust textual data over visual data when inconsistencies arise, leading to significant performance drops under corrupted text and raising safety concerns. We analyze factors influencing this text bias, including instruction prompts, language model size, text relevance, token order, and the interplay between visual and textual certainty. While certain factors, such as scaling up the language model size, slightly mitigate text bias, others like token order can exacerbate it due to positional biases inherited from language models. To address this issue, we explore supervised fine-tuning with text augmentation and demonstrate its effectiveness in reducing text bias. Additionally, we provide a theoretical analysis suggesting that the blind faith in text phenomenon may stem from an imbalance of pure text and multi-modal data during training. Our findings highlight the need for balanced training and careful consideration of modality interactions in VLMs to enhance their robustness and reliability in handling multi-modal data inconsistencies.

Problem

• VLMs disproportionately trust text over visual data.
• Text bias in VLMs leads to performance drops with corrupted text.
• Imbalanced training data may cause VLMs' blind faith in text.

Innovation

• Supervised fine-tuning with text augmentation
• Analysis of factors influencing text bias
• Theoretical analysis of training data imbalance