Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing

2026-07-09

AI Summary

This work addresses the performance degradation of multimodal large language models during inference when privileged modalities available during training become inaccessible. To mitigate this limitation, the authors propose the Mixture of Probes (MoP) framework, which introduces structured probes into intermediate layers of a shared encoder. By integrating a modality-disentanglement loss with a novel MoP-X training strategy, MoP effectively separates and transfers both modality-specific and modality-agnostic knowledge while preventing probe collapse. This approach overcomes the constraints of conventional methods that rely solely on final-layer alignment. Extensive experiments across eight tasks and four modalities demonstrate that MoP achieves up to a 65% relative performance improvement over strong baselines, substantiating the significant benefit of leveraging privileged modalities during training to enhance single-modality inference.

Abstract

Multimodal Large Language Models (MLLMs) are typically designed under the assumption that all modalities available during training will also be accessible at inference. However, many real-world settings violate this assumption, requiring models to operate under a privileged modality setting, where auxiliary modalities are available only during training. While these modalities contain valuable information, existing MLLMs largely fail to leverage them effectively, as they treat modalities as interchangeable inputs rather than sources of complementary supervision. We propose Mixture of Probes (MoP), a novel framework that disentangles modality-specific and modality-general signals within the MLLM, allowing the model to preserve modality-dependent structure while learning transferable representations across modalities. At its core, MoP achieves this through a structured probing mechanism that extracts and organizes information from intermediate representations of a shared modality encoder, rather than relying only on final-layer alignment as done in existing MLLMs. To support this disentanglement, we further introduce MoP Cross-modal Training (MoP-X), a training strategy for MoP centered around a probe disentanglement loss that prevents probe collapse and encourages cross-modal learning. We evaluate MoP across two domains spanning eight tasks and four modalities under a comprehensive evaluation protocol tailored to the privileged modality setting, where each modality is independently treated as the sole input at inference time. MoP consistently outperforms strong MLLM baselines, achieving up to 65% relative improvement, demonstrating that auxiliary modalities, even when unavailable at inference, can provide substantial gains when effectively leveraged during training. Code, model checkpoints, and evaluation protocols will be made available at https://github.com/Sony/MoP.

Problem

• Multimodal Large Language Models
• Privileged Modality
• Modality Disentanglement
• Cross-modal Learning
• Auxiliary Modalities

Innovation

• Mixture of Probes
• privileged modality
• modality disentanglement
• cross-modal learning
• intermediate representation probing