PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment

2026-08-19

AI Summary

为解决多模态偏好优化中的视觉不敏感问题，提出感知增强对齐直接偏好优化(PEA-DPO)方法，通过利用视觉偏好信号来提高模型对视觉上下文的敏感性。

Abstract

Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its adaptation to multimodal settings remains unexplored. Through representational analysis, we identify a key limitation in multimodal preference optimization, which we term visual insensitivity: models often fail to distinguish between images and those with critical visual context removed. Our theoretical analysis further uncovers two manifestations of this problem, namely Across-Image Insensitivity and Within-Image Insensitivity. To address these challenges, we propose Perception-Enhanced Alignment DPO (PEA-DPO), a framework for multimodal LLMs alignment, which explicitly leverages visual preference signals to overcome visual insensitivity. We further provide a theoretical analysis demonstrating that PEA-DPO provably mitigates both failure modes. Empirical results demonstrate that PEA-DPO enhances sensitivity to visual context while preserving the language modeling capacity of the base model. Evaluations across three hallucination benchmarks using MLLMs of varying scales show that PEA-DPO effectively mitigates visual insensitivity, achieves stronger multimodal alignment, and substantially reduces hallucinations.

Problem

• multimodal
• visual insensitivity
• direct preference optimization
• large language models

Innovation

• Perception-Enhanced Alignment
• Visual Insensitivity
• Multimodal LLMs
• Direct Preference Optimization (DPO)
• Across-Image Insensitivity