Mitigating Multimodal Hallucination via Phase-wise Self-reward

2026-04-20

AI Summary

This work addresses the challenge of multimodal hallucinations in large vision-language models (LVLMs), which often generate outputs inconsistent with visual inputs. Existing approaches rely on large-scale annotated data or static post-processing, limiting their ability to dynamically intervene during generation. The study reveals, for the first time, that multimodal hallucinations predominantly emerge during the early semantic generation phase, exhibiting a distinct dynamic pattern. To exploit this insight, the authors propose Phase-Sensitive Reward Decoding (PSRD), a decoding mechanism that enables online, precise intervention without external supervision during inference. PSRD employs a lightweight reward model to distill hallucination-guiding signals, achieving a 50.0% reduction in hallucination rate on LLaVA-1.5-7B. Evaluated across four prominent LVLMs and five benchmarks, PSRD consistently outperforms existing post-hoc methods while preserving generation quality and inference efficiency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Vision-Language Models (LVLMs) still struggle with vision hallucination, where generated responses are inconsistent with the visual input. Existing methods either rely on large-scale annotated data for fine-tuning, which incurs massive computational overhead, or employ static post-hoc strategies that overlook the dynamic nature of hallucination emergence. To address these, we introduce a new self-rewarding framework, enabling dynamic hallucination mitigation at inference time without external supervision. On the empirical side, we reveal that visual hallucination exhibits phase-wise dynamic patterns, peaking at the onset of each semantic phase. Drawing on these insights, we propose \textbf{PSRD} (\textbf{Phase-wise \textbf{S}elf-\textbf{R}eward \textbf{D}ecoding) for online hallucination correction guided by phase-wise self-reward signals. To reduce the cost of repeated self-evaluation during decoding, we distill the hallucination guidance signal from LVLMs into a lightweight reward model. The reward model subsequently provides on-the-fly guidance for targeted intervention during the decoding process, enabling precise hallucination suppression. The proposed PSRD significantly reduces the hallucination rate of LLaVA-1.5-7B by 50.0% and consistently outperforms existing post-hoc methods across five hallucination evaluation benchmarks for four LVLMs. Further analysis confirms that PSRD effectively mitigates hallucination propagation and achieves a highly controllable trade-off between strong performance and inference efficiency.

Problem

• multimodal hallucination
• vision-language models
• hallucination mitigation
• visual hallucination
• LVLMs

Innovation

• Phase-wise Self-Reward
• Multimodal Hallucination
• Vision-Language Models
• Inference-time Mitigation
• Reward Distillation