Context Blindness in DPO: Mitigating Object Hallucination in MLLMs via Context-Calibrated Preference Optimization

2026-08-12

AI Summary

This work addresses the prevalent issue of object hallucination in multimodal large language models, where generated content often conflicts with visual inputs. To mitigate this, the authors propose a context-calibrated preference optimization approach that explicitly enhances the model’s utilization of relevant visual context. Central to this method is a newly designed Contextual Preference Gain (CPG) metric, which is maximized while preserving the original preference ordering. Building upon the Direct Preference Optimization (DPO) framework, the authors construct context-aware preference data and introduce the C²-DPO algorithm to effectively alleviate context-blind generation. Experimental results demonstrate that the proposed method significantly reduces hallucination rates across multiple benchmarks—achieving a 36% relative reduction on Object HalBench for Qwen2-VL-Instruct-2B—without compromising general reasoning capabilities.

Abstract

Multimodal large language models (MLLMs) have made rapid progress, yet they still exhibit object hallucination, generating plausible but incorrect descriptions that are inconsistent with the visual input. Direct Preference Optimization (DPO) mitigates this by training models to prefer non-hallucinated responses over hallucinated ones, and recent efforts further enrich the preference data with relevant context. However, it remains unclear whether DPO actually leverages such context. To investigate this, we propose Contextual Preference Gain (CPG), a simple metric that measures how much a model's preference strengthens when relevant context is provided. We find that higher CPG consistently corresponds to lower hallucination, yet standard DPO and its variants exhibit only limited CPG, indicating that they underutilize contextual information and thus remain prone to hallucination. To address this, we propose Context-Calibrated DPO (C$^2$-DPO), which directly maximizes CPG while preserving the original preference ordering. Across multiple benchmarks, C$^2$-DPO substantially reduces hallucination without compromising general reasoning, relatively reducing the Object HalBench hallucination rate of Qwen2-VL-Instruct-2B by 36%. Code is available at https://github.com/mlvlab/C2-DPO

Problem

• object hallucination
• multimodal large language models
• context blindness
• preference optimization
• visual grounding

Innovation

• Context-Calibrated DPO
• Object Hallucination
• Contextual Preference Gain
• Multimodal LLMs
• Preference Optimization