MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs

2026-05-14

AI Summary

This work addresses the hallucination problem in large vision-language models (LVLMs), which often arises from cross-modal attention biases that lead to outputs inconsistent with visual inputs, thereby undermining model reliability. To mitigate this issue, the authors propose the MHSA framework—the first approach to extend cross-modal attention mechanisms from hallucination detection to active mitigation. MHSA employs a lightweight attention correction strategy that enhances output consistency without modifying the model’s parameters. Specifically, it utilizes a three-layer MLP generator to dynamically refine attention distributions under the joint guidance of a DHCP discriminator and intrinsic signals from the LVLM itself. Experimental results demonstrate that MHSA effectively suppresses both discriminative and generative hallucinations across multiple datasets and LVLM architectures, significantly improving model trustworthiness.

Abstract

Large vision-language models (LVLMs) have achieved remarkable performance across diverse multimodal tasks, yet they continue to suffer from hallucinations, generating content that is inconsistent with the visual input. Prior work DHCP (Detecting Hallucinations by Cross-modal Attention Pattern) has explored hallucination detection from the perspective of cross-modal attention, but does not address hallucination mitigation. In this paper, we propose MHSA (Mitigating Hallucinations via Steered Attention), a lightweight framework that mitigates hallucinations by learning to correct cross-modal attention patterns in LVLMs. MHSA trains a simple three-layer MLP generator to produce corrected attention, guided by supervisory signals from the DHCP discriminator and the LVLM itself. During inference, MHSA mitigates both discriminative and generative hallucinations across various datasets and LVLMs by simply replacing the original cross-modal attention with the corrected one, without modifying any LVLM parameters. By extending cross-modal attention mechanisms from hallucination detection to hallucination mitigation, MHSA offers a novel perspective on hallucination research in LVLMs and helps enhance their reliability.

Problem

• hallucinations
• large vision-language models
• cross-modal attention
• multimodal tasks

Innovation

• hallucination mitigation
• steered attention
• cross-modal attention
• large vision-language models
• lightweight framework