Groc-PO: Grounded Context Preference Optimization for Truthful Multimodal LLMs

2026-07-15

AI Summary

This work addresses the prevalent issue of visual hallucinations and unfaithful reasoning in multimodal large language models, which often stems from insufficient supervision during early grounding stages and leads to error propagation across phases. To mitigate this, the authors propose the Groc-PO framework, which introduces a novel refinement of preference optimization into three distinct stages: object grounding, contextual grounding, and grounded reasoning. They construct a multi-stage preference dataset, GCPD, and extend Direct Preference Optimization (DPO) to enable stage-wise alignment training. By applying explicit grounding supervision to directly intervene at the source of errors, Groc-PO significantly outperforms standard DPO and other strong baselines in hallucination suppression, reasoning faithfulness, and overall reliability, thereby demonstrating the effectiveness of multi-stage preference modeling for trustworthy multimodal reasoning.

Abstract

Despite the rapid progress of Multimodal Large Language Models (MLLMs), they still suffer from untruthfulness issues, such as visual hallucinations, content fabrication, and unfaithful reasoning, which substantially undermine their faithfulness and practical utility. Alignment methods based on human preference, such as Direct Preference Optimization (DPO), have been widely adopted to address these issues. However, multimodal reasoning errors often propagate across stages, and final-answer errors can often be traced to mistakes in early grounding stages, yet standard DPO typically applies preference optimization at the final-answer level. This credit-assignment challenge means that supervision for early grounding stages is indirect rather than stage-specific, making it difficult to suppress error propagation arising from grounding drift and context inconsistency. To address this, we propose Grounded Context Preference Optimization (Groc-PO), a grounded preference optimization framework for MLLMs. We further construct the Grounded Context Preference Dataset (GCPD), organizing multi-stage preference samples around three stages of Object Grounding, Contextual Grounding, and Grounded Reasoning, to capture the formation, integration, and utilization of grounded context. By introducing more explicit preference supervision over multiple grounded stages, Groc-PO strengthens context-dependent reasoning and mitigates cross-stage error propagation. Extensive experiments show that, compared with standard DPO and other strong baselines, Groc-PO achieves improved performance in hallucination mitigation, faithful reasoning, and overall reliability, supporting the value of more explicit grounded supervision for trustworthy multimodal reasoning.

Problem

• Multimodal Large Language Models
• visual hallucinations
• preference optimization
• grounding drift
• error propagation

Innovation

• Grounded Context Preference Optimization
• Multimodal LLMs
• Preference Optimization
• Error Propagation Mitigation
• Faithful Reasoning