Rethinking Gating Mechanism in Sparse MoE: Handling Arbitrary Modality Inputs with Confidence-Guided Gate

2025-05-26

AI Summary

Sparse Mixture-of-Experts (SMoE) models suffer from performance degradation and poor generalization under multimodal missing-input scenarios. Method: We propose Conf-SMoE, a confidence-guided two-stage imputation and gating architecture. We first theoretically identify and analyze the “expert collapse” phenomenon—where routing collapses to a subset of experts—and design a novel decoupled gating mechanism that separately computes routing scores and task-specific confidence, eliminating the need for load-balancing loss to mitigate collapse. Combined with modality-adaptive imputation, Conf-SMoE enables robust handling of missing inputs. Results: Extensive experiments across four real-world datasets and three missing patterns demonstrate that Conf-SMoE significantly improves accuracy and robustness, achieves more stable multimodal fusion, and generalizes strongly to arbitrary modality missingness. Furthermore, Gaussian/Laplace gating consistency analysis provides theoretical interpretability of the gating behavior.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Multi-modal Vision
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Effectively managing missing modalities is a fundamental challenge in real-world multimodal learning scenarios, where data incompleteness often results from systematic collection errors or sensor failures. Sparse Mixture-of-Experts (SMoE) architectures have the potential to naturally handle multimodal data, with individual experts specializing in different modalities. However, existing SMoE approach often lacks proper ability to handle missing modality, leading to performance degradation and poor generalization in real-world applications. We propose Conf-SMoE to introduce a two-stage imputation module to handle the missing modality problem for the SMoE architecture and reveal the insight of expert collapse from theoretical analysis with strong empirical evidence. Inspired by our theoretical analysis, Conf-SMoE propose a novel expert gating mechanism by detaching the softmax routing score to task confidence score w.r.t ground truth. This naturally relieves expert collapse without introducing additional load balance loss function. We show that the insights of expert collapse aligns with other gating mechanism such as Gaussian and Laplacian gate. We also evaluate the proposed method on four different real world dataset with three different experiment settings to conduct comprehensive the analysis of Conf-SMoE on modality fusion and resistance to missing modality.

Problem

• Handling missing modalities in multimodal learning scenarios
• Preventing expert collapse in Sparse Mixture-of-Experts architectures
• Improving generalization with confidence-guided gating mechanism

Innovation

• Two-stage imputation module for missing modalities
• Confidence-guided expert gating mechanism
• Detaches softmax routing to task confidence