Global Context or Local Detail? Adaptive Visual Grounding for Hallucination Mitigation

2026-04-27

AI Summary

This work addresses the prevalent issue of object hallucination in vision-language models, which often stems from their overreliance on linguistic priors. To enhance visual faithfulness without requiring additional training, the authors propose a Positive-Negative Decoding (PND) framework that dynamically intervenes in the generation process during inference. The positive path amplifies salient visual evidence, while the negative path constructs counterfactual representations to suppress misleading object features. This approach uniquely identifies and mitigates attention deficiencies within the model through multi-layer attention mechanisms, counterfactual feature degradation, and cross-model compatibility. Evaluated on standard benchmarks—including POPE, MME, and CHAIR—the method achieves state-of-the-art performance, improving accuracy by up to 6.5%, substantially reducing hallucinations, and enhancing fine-grained visual descriptions across diverse architectures such as LLaVA, InstructBLIP, InternVL, and Qwen-VL.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Vision-Language Models (VLMs) are frequently undermined by object hallucination--generating content that contradicts visual reality--due to an over-reliance on linguistic priors. We introduce Positive-and-Negative Decoding (PND), a training-free inference framework that intervenes directly in the decoding process to enforce visual fidelity. PND is motivated by our key finding of a critical attention deficit in VLMs, where visual features are empirically under-weighted. Our framework corrects this via a dual-path contrast: The positive path amplifies salient visual evidence using multi-layer attention to encourage faithful descriptions, directly counteracting the attention deficit. Simultaneously, the negative path identifies and degrades the core object's features to create a strong counterfactual, which penalizes ungrounded, prior-dominant generation. By contrasting the model's outputs from these two perspectives at each step, PND steers generation towards text that is not just linguistically probable, but visually factual. Extensive experiments on benchmarks like POPE, MME, and CHAIR show that PND achieves state-of-the-art performance with up to 6.5% accuracy improvement, substantially reducing object hallucination while also enhancing descriptive detail--all without requiring any model retraining. The method generalizes effectively across diverse VLM architectures including LLaVA, InstructBLIP, InternVL, and Qwen-VL.

Problem

• object hallucination
• vision-language models
• visual grounding
• linguistic priors
• visual fidelity

Innovation

• Positive-and-Negative Decoding
• visual grounding
• object hallucination
• attention deficit
• training-free inference