MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignment

2026-04-23

AI Summary

This work addresses the limitations of existing general-purpose multimodal retrieval methods, which either suffer from early fusion strategies that overly prioritize textual cues at the expense of visual information or late fusion approaches that cause semantically related samples to be distant in the embedding space, leading to modality collapse and semantic misalignment. To overcome these issues, the authors propose MiMIC, a novel framework that integrates multimodal fusion within the decoder and employs robust training strategies—including unimodal mixing and random caption dropping—to achieve balanced modality representation and precise semantic alignment within a unified architecture. Experimental results demonstrate that MiMIC significantly outperforms current baselines on the WebQA+ and EVQA+ benchmarks, exhibiting particularly strong robustness in weakly supervised scenarios such as missing image captions.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Universal Multimodal Retrieval (UMR) aims to map different modalities (e.g., visual and textual) into a shared embedding space for multi-modal retrieval. Existing UMR methods can be broadly divided into two categories: early-fusion approaches, such as Marvel, which projects visual features into the language model (LM) space for integrating with text modality, and late-fusion approaches, such as UniVL-DR, which encode visual and textual inputs using separate encoders and obtain fused embeddings through addition. Our pilot study reveals that Marvel exhibits visual modality collapse, which is characterized by the model's tendency to disregard visual features while depending excessively on textual cues. In contrast, although UniVL-DR is less affected by this issue, it is more susceptible to semantic misalignment, where semantically related content is positioned far apart in the embedding space. To address these challenges, we propose MiMIC, which introduces two key innovations: (1) a fusion-in-decoder architecture for effective multimodal integration, and (2) robust training through single modality mixin and random caption dropout. Experiments on the WebQA+ and EVQA+ datasets, where image in documents or queries might lack captions, indicate that MiMIC consistently outperforms both early- and late-fusion baselines.

Problem

• visual modality collapse
• semantic misalignment
• universal multimodal retrieval
• multimodal embedding
• modality integration

Innovation

• fusion-in-decoder
• visual modality collapse
• semantic misalignment
• random caption dropout
• multimodal retrieval