Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs

2025-11-27

AI Summary

This study addresses the robustness deficiency of multimodal large language models (MLLMs) under modality conflicts—such as audio-visual inconsistencies or text-based misdirection—revealing their over-reliance on single modalities. To systematically evaluate this vulnerability, we introduce MMA-Bench, the first benchmark explicitly designed for modality conflict assessment. Our method proposes a context-aware modality alignment fine-tuning framework: it constructs test sets using video–task pairs, integrates black-box and white-box interpretability analyses, and designs a modality alignment loss to dynamically calibrate cross-modal attention weights during inference. Empirical results demonstrate substantial improvements in reasoning accuracy and multimodal grounding under contradictory inputs across diverse open- and closed-source MLLMs. The approach generalizes effectively across architectures and establishes a new paradigm for robust multimodal reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Despite remarkable advancements in Multimodal Large Language Models (MLLMs), a fundamental question remains: are MLLMs robust to contradicting modalities? To rigorously study this, we introduce MMA-Bench comprising videos and tasks that probe a model's reliance on specific modalities. Using black-box and white-box interpretability techniques, we provide a critical analysis of the brittleness of both open- and closed-sourced MLLMs. We show that current MLLMs struggle under misaligned audio-visual pairs and simple misleading text, thereby lacking robust multi-modal reasoning. Building on these findings, we propose a modality alignment tuning strategy to teach the model when to prioritize, leverage, or ignore specific modality cues. Through extensive experiments and analysis, we show that our alignment tuning yields demonstrably stronger multimodal grounding. This work provides both interpretability tools and a clear path toward developing MLLMs with intrinsically reliable cross-modal reasoning. Code and dataset will be publicly available.

Problem

• Assess MLLM robustness to conflicting multimodal inputs
• Propose modality alignment tuning for reliable cross-modal reasoning
• Provide interpretability tools for analyzing multimodal integration brittleness

Innovation

• Proposed modality alignment tuning strategy
• Introduced MMA-Bench dataset for evaluation
• Used black-box and white-box interpretability techniques