Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding

2026-06-25

AI Summary

This work addresses the susceptibility of large vision-language models to language priors during inference, which often leads to object hallucinations detached from visual evidence. The authors propose Fox, a novel framework that, for the first time, identifies hallucinations as stemming from a structural mismatch between attention heads critical to decision-making and actual visual evidence. Fox introduces a training-free causal intervention mechanism: it employs a visual attention entropy probe to locate high-risk attention heads, applies logit saturation to sever pathological shortcuts, and incorporates a conflict-aware gating mechanism in decoding to jointly preserve generation fluency and faithfulness. Without compromising linguistic richness, Fox substantially outperforms the state-of-the-art SID method by 29.1% in hallucination mitigation.

Abstract

Large Vision-Language Models (LVLMs) exhibit sophisticated reasoning but remain susceptible to object hallucination. Deviating from the prevailing attention intensity assumption, we reveal a deeper dynamic structural misalignment: hallucination is triggered at decision-critical steps where specific attention heads, acting as risky mediators, decouple from visual evidence to lock onto language priors. This establishes a pathological shortcut that bypasses visual grounding. To dismantle this, we propose Fox (Faithfulness and Observational-flow via eXpression-rectification), a training-free inference-time framework. Fox diagnoses structural misalignment using a visual attention entropy probe to localize risky mediators unsupervisedly. We then execute a targeted causal intervention via numerical logit saturation to physically sever the shortcut path. Finally, a conflict-gated cooperative decoding strategy reconciles interventional faithfulness with observational fluency. Extensive experiments demonstrate that Fox achieves SOTA performance, outperforming SID by 29.1% while preserving linguistic richness. Code is available at https://github.com/Cc2021start/Fox.

Problem

• object hallucination
• visual grounding
• pathological shortcut
• attention misalignment
• faithful decoding

Innovation

• causal intervention
• visual grounding
• object hallucination
• attention entropy
• training-free decoding