ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities

2026-07-07

AI Summary

This work addresses the challenge of missing modalities in multimodal federated learning by proposing ProMoE-FL, a novel framework that avoids reliance on external public data or simplistic synthesis based solely on available modalities. The method introduces, for the first time, a clinically meaningful cross-institutional modality prior prototype bank and designs a prototype- and modality-index-conditioned mixture-of-experts (MoE) routing mechanism to enable direction-aware dynamic feature synthesis. By incorporating client-aware prototype modeling, ProMoE-FL significantly enhances the semantic consistency and robustness of missing modality reconstruction. Extensive experiments on four chest X-ray datasets demonstrate that ProMoE-FL consistently outperforms state-of-the-art methods under both homogeneous and heterogeneous settings.

Abstract

In this paper, we address the problem of multimodal federated learning with missing modality. Existing methods utilize an additional public dataset or perform naive feature synthesis that is based solely on the available modality. To address these limitations, we propose ProMoE-FL, a Prototype-conditioned Mixture-of-Experts framework for robust missing-modality feature synthesis in multimodal federated learning. ProMoE-FL builds a global client-aware prototype bank that captures clinically meaningful modality priors across institutions. Our Mixture of Experts is conditioned on these prototypes and modality indices to enable direction-aware expert routing for dynamically synthesizing missing features. We perform extensive quantitative and qualitative evaluations on four public chest X-ray datasets (MIMIC-CXR, NIH Open-I, PadChest, and CheXpert) and demonstrate that ProMoE-FL consistently outperforms state-of-the-art methods in both homogeneous as well as the more challenging heterogeneous settings.

Problem

• multimodal federated learning
• missing modalities
• feature synthesis
• Mixture of Experts
• prototype conditioning

Innovation

• Prototype-conditioned
• Mixture of Experts
• Multimodal Federated Learning
• Missing Modality
• Feature Synthesis