FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models

2026-06-28

AI Summary

This work addresses the prevalent issue of hallucinations in large vision-language models, which often arise when language priors dominate over visual evidence. Through a detailed analysis of information flow within Transformer layers, the study identifies the feed-forward network (FFN) as a primary source of such language priors—a finding reported for the first time. To mitigate this, the authors propose FADE, a training-free decoding intervention that attenuates FFN outputs during inference to suppress the overpowering influence of language priors. Inspired by contrastive decoding, FADE effectively preserves factual consistency with visual inputs while maintaining computational efficiency. Extensive experiments demonstrate that FADE consistently reduces hallucination rates and improves factuality across multiple benchmarks—including POPE, CHAIR, and MME—for prominent models such as LLaVA-1.5, mPLUG-Owl2, and InstructBLIP.

Abstract

Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucination, generating content inconsistent with the input image. Recent studies attribute this to the dominance of language priors over visual inputs and employ contrastive decoding methods to mitigate this dominance, but the mechanistic origin remains unexplored. We investigate the information flow through each transformer layer and find that attention modules consistently aggregate visual evidence, while FFN modules at critical layers act as the source of language priors. These priors can override visual evidence, causing correct predictions in intermediate layers to drift toward incorrect outputs. Based on this insight, we propose FADE (FFN Attenuation for DEcoding), a training-free method that attenuates FFN outputs to reduce language-prior dominance. Evaluations on POPE, CHAIR, and MME benchmarks across LLaVA-1.5, mPLUG-Owl2, and InstructBLIP show that FADE effectively mitigates hallucinations while preserving inference efficiency.

Problem

• hallucination
• language priors
• vision-language models
• visual grounding
• model reliability

Innovation

• hallucination mitigation
• language-prior dominance
• FFN attenuation
• vision-language models
• training-free decoding