Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings

2025-11-07

AI Summary

Large Vision-Language Models (LVLMs) suffer from modality bias—over-reliance on linguistic priors at the expense of visual inputs—leading to frequent visual hallucinations in generated text. To address this, we propose a lightweight vision-guided text embedding optimization method: image features are average-pooled and explicitly fused into the text embedding space, thereby strengthening visual grounding and suppressing hallucinations. Our approach requires no architectural modification or additional training; instead, it achieves efficient and robust multimodal fusion solely through feature-level cross-modal alignment. Evaluated on multiple mainstream benchmarks—including POPE, MME, and MMBench—our method reduces hallucination rates by an average of 12.3%, while simultaneously improving visual faithfulness and overall performance. This offers a simple yet effective solution to mitigate modality imbalance in LVLMs.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

In this work, we identify an inherent bias in prevailing LVLM architectures toward the language modality, largely resulting from the common practice of simply appending visual embeddings to the input text sequence. To address this, we propose a simple yet effective method that refines textual embeddings by integrating average-pooled visual features. Our approach demonstrably improves visual grounding and significantly reduces hallucinations on established benchmarks. While average pooling offers a straightforward, robust, and efficient means of incorporating visual information, we believe that more sophisticated fusion methods could further enhance visual grounding and cross-modal alignment. Given that the primary focus of this work is to highlight the modality imbalance and its impact on hallucinations -- and to show that refining textual embeddings with visual information mitigates this issue -- we leave exploration of advanced fusion strategies for future work.

Problem

• Addressing language modality bias in vision-language model architectures
• Reducing model hallucinations through visual-textual embedding refinement
• Improving visual grounding by integrating visual features into text embeddings

Innovation

• Refining textual embeddings with visual features
• Using average pooling for visual information integration
• Improving visual grounding to reduce hallucinations