Information Router for Mitigating Modality Dominance in Vision-Language Models

2026-04-17

AI Summary

This work addresses the modality dominance problem in vision-language models, where imbalanced information density or signal-to-noise ratios across modalities lead predictions to over-rely on a single modality. To mitigate this, the authors propose a Multimodal Information Routing mechanism (MoIR), which—unlike attention-based approaches—explicitly identifies information-deficient modality tokens at the information level and dynamically injects complementary signals from high-information modalities to enrich the representations of weaker ones. By regulating cross-modal information supply prior to fusion, MoIR enables more balanced and robust multimodal integration. Extensive experiments demonstrate that MoIR consistently improves modality contribution balance, enhances downstream task performance, and increases robustness under modality degradation scenarios across multiple mainstream benchmarks and model architectures.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Vision Language models (VLMs) have demonstrated strong performance across a wide range of benchmarks, yet they often suffer from modality dominance, where predictions rely disproportionately on a single modality. Prior approaches primarily address this issue by steering model's attention allocation, implicitly assuming that all modalities provide sufficient information. However, attention only determines where the model focuses, and cannot enrich information that is missing or ambiguous. In the real world, input modalities often differ in information density and their signal-to-noise ratios. In such cases, simply adjusting model's attention does not resolve the underlying lack of information. In this paper, we propose \textsc{MoIR}: \textit{Multi-modal Information Router}, an information-level fusion method that explicitly reduces information disparity prior to fusion. \textsc{MoIR} identifies less informative tokens and routes complementary information from a stronger modality, constructing information-dense token representations before they are processed by a large language model. By modifying information availability, \textsc{MoIR} enables reliable shifts in modality dominance, even when one modality is degraded. We evaluate \textsc{MoIR} on three widely used multi-modal benchmarks across multiple model backbones. Experimental results show that \textsc{MoIR} consistently demonstrates more balanced modality contribution, and improves robustness and downstream performance, particularly even under modality degradation. These findings demonstrate that explicitly modifying cross-modal information is an effective and complementary strategy for mitigating modality dominance in multi-modal reasoning models.

Problem

• modality dominance
• vision-language models
• information disparity
• multi-modal fusion
• modality degradation

Innovation

• modality dominance
• information routing
• multi-modal fusion
• vision-language models
• information disparity