See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

2026-06-29

AI Summary

This work addresses object hallucination in large vision-language models (LVLMs), a prevalent issue during text generation that existing training-free methods often exacerbate by over-relying on visual inputs at the expense of linguistic fluency. The authors propose Context-Aware Intervention (CAI), a training-free, inference-time mechanism guided by the principle “attend only when necessary.” CAI leverages early-layer representations to identify semantically aligned regions and dynamically triggers fine-grained attention intervention during deep decoding by jointly assessing visual relevance and prediction uncertainty—applying conservative attention shifts precisely when visual grounding degrades and entropy is high. Theoretically, CAI corresponds to the minimal-KL reweighting of attention distributions and optionally integrates contrastive decoding for further debiasing. Experiments demonstrate that CAI consistently outperforms current training-free approaches across multiple LVLM architectures and benchmarks, effectively suppressing hallucinations while preserving language fluency.

Abstract

Large Vision-Language Models (LVLMs) excel at multimodal tasks but remain prone to object hallucinations. Prior training-free remedies often uniformly strengthen visual signals, which may also amplify irrelevant regions and introduce spurious evidence, harming fluency. We propose Context-aware Attention Intervention (CAI), a training-free inference-time mechanism that enforces a see only when needed principle via two-axis selectivity: where to look and when to intervene. At each decoding step, CAI derives token-specific visual relevance from early-layer representations to localize semantically aligned regions, and applies a conservative, entropy- and depth-gated attention tilt only for uncertainty-spiking tokens in deeper layers where visual grounding degrades, leaving confident tokens and irrelevant regions largely unchanged. This targeted intervention strengthens visual grounding while preserving linguistic fluency, and it yields consistent improvements even without contrastive decoding, which remains optional as an auxiliary bias-suppression module. Extensive experiments across multiple LVLM backbones and benchmarks show that CAI achieves state-of-the-art hallucination mitigation, and our analysis characterizes CAI as a KL-minimal attention reweighting with bounded interference under inactive gates or small tilts. Code is available at https://github.com/Iris1946/CAI.

Problem

• hallucination
• Large Vision-Language Models
• visual grounding
• object hallucinations
• multimodal tasks

Innovation

• Context-aware Attention Intervention
• Hallucination Mitigation
• Visual Grounding
• Attention Reweighting
• Training-free Inference