Residual-Guided Expert Specialization for Incomplete Multimodal Learning

2026-06-29

AI Summary

This work addresses the performance degradation of multimodal prediction systems during inference due to missing modalities by proposing the MARS framework. MARS is the first to leverage residual signals between complete and incomplete modality representations to guide expert specialization within a mixture-of-experts architecture. It introduces a feature router that performs efficient, modality-aware routing using only available inputs, alongside a novel residual-feature collaborative routing mechanism and discrepancy-aware noise regularization to enhance model robustness. Evaluated across multiple benchmarks—including CASIA-SURF, CREMA-D, UPMC Food-101, and MCubeS—MARS consistently outperforms existing methods, demonstrating strong efficiency, scalability, and generalizability across diverse backbone networks and tasks.

Abstract

As real-world prediction systems often face missing modalities at inference, incomplete multimodal learning (IML) remains a practical challenge. While prior methods aim to learn representations robust to missing inputs, representations from incomplete modalities inevitably deviate from their full-modality counterparts due to missing evidence. To explicitly leverage these deviations, we propose MARS (Missingness-Aware Residual-guided Specialization), a mixture-of-experts framework that guides expert specialization based on how representations are reshaped by missingness. By contrasting task representations derived from incomplete inputs with their complete counterparts during training, we derive a privileged residual signal that captures this representational gap. The residual signal guides a residual router to assign samples to experts specialized for the corresponding deviation patterns. In parallel, a feature router learns to imitate this routing behavior using only incomplete inputs, enabling deployment without access to full modalities. To mitigate this train-test router gap, we develop a discrepancy-aware noise regularization that adaptively perturbs the residual router's decisions when the feature router deviates, enhancing expert robustness under imperfect imitation. Experiments on multimodal classification (CASIA-SURF, CREMA-D, UPMC Food-101) and segmentation (MCubeS) under missing scenarios show that MARS consistently surpasses baselines while remaining efficient and extensible to diverse backbones and tasks.

Problem

• incomplete multimodal learning
• missing modalities
• representation deviation
• multimodal prediction
• modality missingness

Innovation

• incomplete multimodal learning
• mixture-of-experts
• residual-guided specialization
• missingness-aware routing
• noise regularization