Reducing Hallucination in Vision-Language Models via Stage-wise Preference Optimization under Distribution Shift

2026-05-13

AI Summary

This work addresses the issue of visually inconsistent hallucinations in existing vision-language models caused by joint modeling during autoregressive generation. To mitigate this, the authors propose a staged preference optimization framework that constructs minimally perturbed yet visually inconsistent negative samples to precisely approximate the hallucination boundary. By integrating Direct Preference Optimization (DPO), the method enables targeted training on critical scenarios involving spatial relations, OCR uncertainty, and adversarial false premises. Experimental results demonstrate that this approach significantly reduces hallucination rates and enhances visual grounding consistency, achieving superior generation performance over leading closed-source models on both open-source benchmarks and real-world multimodal tasks.

Abstract

Hallucination remains a fundamental challenge in vision-language models (VLMs), where autoregressive generation may produce linguistically plausible yet physically inconsistent or visually ungrounded responses due to likelihood maximization under joint probabilistic modeling. We propose a stage-wise preference optimization framework for hallucination reduction through targeted multimodal data construction. Rather than directly optimizing on generic instruction-following data, our approach progressively constructs hallucination-focused preference pairs near known failure boundaries. The framework emphasizes ambiguous spatial orientation, object relationships, OCR uncertainty, and adversarial false-premise training. Hallucinated negatives are generated through minimally perturbed yet visually inconsistent alternatives, enabling Direct Preference Optimization (DPO) to better separate grounded reasoning from plausible hallucination. Experiments on open-source benchmarks and real-world multimodal evaluation scenarios demonstrate improved grounding consistency, reduced hallucination, and more informative grounded responses. Cross-model qualitative evaluation further shows that the proposed multimodal LLM DPO framework produces more visually grounded responses than several frontier proprietary VLMs, such as in ambiguous spatial reasoning and adversarial false-premise settings. The results suggest that hallucination may arise not only from limited model capacity, but also from inherent tendencies of autoregressive probabilistic generation to favor linguistically plausible continuations under weak visual grounding. Future work may explore physical consistency modeling, uncertainty-aware multimodal reasoning, and architectural alternatives beyond standard autoregressive decoding.

Problem

• Hallucination
• Vision-Language Models
• Visual Grounding
• Autoregressive Generation
• Multimodal Reasoning

Innovation

• Stage-wise Preference Optimization
• Vision-Language Models
• Hallucination Reduction
• Direct Preference Optimization (DPO)
• Multimodal Grounding