Multimodal Federated Learning With Missing Modalities through Feature Imputation Network

2025-05-26

AI Summary

In medical multimodal federated learning, clinical data often suffer from modality missingness due to cost, privacy, and technical constraints. Existing approaches—relying on real or synthetic data imputation—face challenges including data acquisition difficulty, high generative overhead, and significant fidelity degradation. To address this, we propose a lightweight low-dimensional feature translator that enables collaborative reconstruction of missing-modality bottleneck-layer features across clients without sharing raw data. This introduces the novel paradigm of *feature-space modality completion*, circumventing the computational burden and fidelity risks inherent in high-dimensional medical image generation while supporting robust training across heterogeneous clients. Our method integrates feature distillation, cross-modal bottleneck mapping, and FedAvg-based optimization. Extensive experiments on MIMIC-CXR, NIH Open-I, and CheXpert demonstrate consistent and significant improvements over state-of-the-art baselines. The implementation is publicly available.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multimodal federated learning holds immense potential for collaboratively training models from multiple sources without sharing raw data, addressing both data scarcity and privacy concerns, two key challenges in healthcare. A major challenge in training multimodal federated models in healthcare is the presence of missing modalities due to multiple reasons, including variations in clinical practice, cost and accessibility constraints, retrospective data collection, privacy concerns, and occasional technical or human errors. Previous methods typically rely on publicly available real datasets or synthetic data to compensate for missing modalities. However, obtaining real datasets for every disease is impractical, and training generative models to synthesize missing modalities is computationally expensive and prone to errors due to the high dimensionality of medical data. In this paper, we propose a novel, lightweight, low-dimensional feature translator to reconstruct bottleneck features of the missing modalities. Our experiments on three different datasets (MIMIC-CXR, NIH Open-I, and CheXpert), in both homogeneous and heterogeneous settings consistently improve the performance of competitive baselines. The code and implementation details are available at: https://github.com/bhattarailab/FedFeatGen

Problem

• Address missing modalities in multimodal federated learning
• Overcome data scarcity and privacy in healthcare AI
• Replace costly generative models with lightweight feature imputation

Innovation

• Lightweight feature translator for missing modalities
• Improves performance in homogeneous and heterogeneous settings
• Low-dimensional bottleneck feature reconstruction