Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization

2025-03-23

AI Summary

Multimodal large language models (MLLMs) often suffer from modality bias due to imbalanced modality dependence, resulting in weak cross-modal alignment and high hallucination rates. To address this, we propose a noise-aware preference optimization framework. First, we construct RLAIFVBias—the first preference dataset explicitly designed for modality bias mitigation. Second, we develop a robust DPO algorithm incorporating negative Box-Cox transformation to enable adaptive adjustment to varying noise levels. Third, we introduce automated modality perturbation injection and dynamic noise weighting to enhance robustness. Our method preserves generation quality while significantly alleviating modality bias: it improves cross-modal alignment by +4.2% on benchmarks including MMBench and OCRVQA, reduces hallucination rates by −18.7%, and enhances response relevance and factual consistency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Multimodal Large Language Models excel in various tasks, yet often struggle with modality bias, where the model tends to rely heavily on a single modality and overlook critical information in other modalities, which leads to incorrect focus and generating irrelevant responses. In this paper, we propose using the paradigm of preference optimization to solve the modality bias problem, including RLAIFVBias, a debiased preference optimization dataset, and a Noise Aware Preference Optimization algorithm. Specifically, we first construct the dataset by introducing perturbations to reduce the informational content of certain modalities, compelling the model to rely on a specific modality when generating negative responses. To address the inevitable noise in automatically constructed data, we combine the noise robust Mean Absolute Error with the Binary Cross Entropy in Direct Preference Optimization by a negative Box Cox transformation, and dynamically adjust the algorithm noise robustness based on the evaluated noise levels in the data. Extensive experiments validate our approach, demonstrating not only its effectiveness in mitigating modality bias but also its significant role in minimizing hallucinations.

Problem

• Addressing modality bias in Multimodal Large Language Models
• Reducing noise impact in debiased preference optimization
• Mitigating hallucinations through noise-aware algorithms

Innovation

• Noise-Aware Preference Optimization algorithm
• RLAIFVBias debiased dataset
• Dynamic noise robustness adjustment