ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs

2026-06-29

AI Summary

This work addresses the susceptibility of multimodal large language models (MLLMs) to hallucination during decoding, which stems from degradation in cross-modal text-image attention mechanisms. The authors propose ADAPT, a novel framework that intervenes in hallucination by explicitly regulating internal attention dynamics from the perspective of their temporal evolution. ADAPT integrates three key mechanisms: establishing early visual anchors, online correction of attention drift, and visual-attention-guided preference alignment via Direct Preference Optimization (DPO). By combining cross-attention visualization analysis with inference-time supervision, the method reduces hallucination rates by 40%–60% across multiple established benchmarks while preserving strong general multimodal capabilities, achieving state-of-the-art performance.

Abstract

Multimodal Large Language Models (MLLMs) are critically hampered by hallucination, generating content inconsistent with the provided image. In this paper, we identify an internal signature of hallucination: progressive degradation of text-to-image cross-attention during generation, leading to specific failure patterns like unfocused or biased attention. Existing mitigation strategies are largely outcome-driven and do not explicitly target this failure mode. To address this problem, we propose ADAPT (Attention Dynamics Alignment with Preference Tuning), an attention-based framework that intervenes directly on text-to-image cross-attention dynamics. We propose ADAPT with three key contributions: a cross-attention visual anchor refined from early decoding to provide stable spatial grounding, an attention-supervised inference mechanism that detects and corrects attention drift online, and a Visual Attention Guidance DPO that aligns preferences toward visually grounded responses. Experiments show that each component of ADAPT contributes to hallucination reduction, and the full framework achieves new best results across multiple hallucination benchmarks, reducing hallucination rates by 40%-60% across mainstream backbones while preserving general multimodal capabilities. Our work provides an attention-based perspective on mitigating hallucinations by exploring the model's internal text-to-image cross-attention behaviors. Code is available at https://github.com/yao-ustc/ADAPT

Problem

• hallucination
• multimodal large language models
• cross-attention
• attention degradation
• visual grounding

Innovation

• cross-attention dynamics
• visual grounding
• hallucination mitigation
• preference tuning
• attention supervision