FM$^2$: Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

2026-07-14

AI Summary

This work addresses the dual challenges of modality heterogeneity—encompassing both overlapping and non-overlapping modalities—and privacy preservation in federated medical imaging. To this end, the authors propose FM², a novel framework that trains a medical-specific backbone from scratch under a federated setting, optionally incorporating biomedical pre-trained encoders. FM² equips each client with a dual Mixture-of-Experts module operating at both category and domain levels. A key innovation is the introduction of a Heterogeneous Modality Alignment (HMA) regularizer, complemented by GPT-4o-generated image captions, to enable semantic alignment and cross-modal knowledge transfer without requiring shared modalities across clients. Extensive experiments demonstrate that FM² significantly outperforms existing methods on the authors’ newly established MIMH benchmark and real-world medical VQA datasets, achieving state-of-the-art performance in classification, caption-augmented learning, and cross-modal generalization tasks.

Abstract

Building foundation models for medical imaging requires pooling data across institutions, yet privacy regulations prohibit centralized aggregation. Existing Federated Foundation Models either fine-tune natural-image models with poor medical-domain transfer, or train from scratch within a single modality, lacking the flexibility to unify tasks. We identify an under-explored challenge, Imaging Modality Heterogeneity, where clients operate under two structural regimes: Overlapped (shared modalities with heterogeneous label distributions) and Non-overlapped (fully disjoint modalities per client). We propose FM$^2$, a unified framework that trains the core backbone from scratch to preserve medical domain fidelity while optionally incorporating biomedical pretrained encoders for vision-language alignment. FM$^2$ equips each client with dual Mixture-of-Experts modules (a Class-wise MoE for personalized category knowledge and a Domain-wise MoE for shared cross-modality representations), coupled with a Heterogeneous Modality Alignment (HMA) regularizer that explicitly aligns modality-specific expert parameters, admitting provable $O(1/\sqrt{T})$ convergence and generalization guarantees. FM$^2$ further incorporates Caption-Enhanced Learning (CEL), where locally retained GPT-4o-generated captions serve as a textual semantic bridge enabling representation transfer across clients with disjoint modalities, and demonstrates extensibility to Federated Medical VQA. Experiments on our MIMH benchmark (classification and CEL) and real-world medical VQA datasets confirm consistent superiority over state-of-the-art federated baselines and strong out-of-modality generalization across all three tasks.

Problem

• Federated Learning
• Foundation Models
• Medical Imaging
• Modality Heterogeneity
• Privacy Preservation

Innovation

• Federated Foundation Models
• Modality Heterogeneity
• Mixture-of-Experts
• Heterogeneous Modality Alignment
• Caption-Enhanced Learning