Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation

2026-04-22

AI Summary

This work addresses the challenge of hallucination in large vision-language models, which undermines output reliability while existing mitigation strategies often compromise general generative capabilities. The authors propose MPD, a two-stage framework that, for the first time, enables semantic-aware precise disentanglement of hallucinatory components and facilitates interpretable, selective updating of model parameters most associated with hallucination. Notably, this approach incurs no additional computational overhead. Evaluated on LLaVA-Bench and MME benchmarks, MPD reduces hallucination by 23.4% while preserving 97.4% of general generation performance, significantly outperforming current state-of-the-art methods.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Vision-Language Models (LVLMs) exhibit powerful generative capabilities but frequently produce hallucinations that compromise output reliability. Fine-tuning on annotated data devoid of hallucinations offers the most direct solution, while its high computational cost motivates recent representation-based methods, which focus on mitigating hallucinatory components within hidden representations. Though efficient, we empirically observe that these methods degrade general generation capacity due to incomplete extraction of hallucination components and non-selective parameter updates. To address these limitations, we propose MPD, a dual-stage framework for mitigating hallucinations without performance degradation. Specifically, our MPD relies on two essential factors: (1) semantic-aware component disentanglement to extract pure hallucination components, and (2) interpretable parameter updates that selectively modify parameters most relevant to hallucination. Extensive experiments demonstrate that MPD achieves state-of-the-art performance, reducing hallucinations by 23.4\% while maintaining 97.4\% of general generative capability as evaluated on LLaVA-Bench and MME, with no additional computational cost.

Problem

• hallucination
• large vision-language models
• performance degradation
• reliability
• generative capability

Innovation

• hallucination mitigation
• vision-language models
• semantic-aware disentanglement
• selective parameter update
• representation-based methods