Multimodal Federated Learning under Dual-Axis Modality Missingness

2026-08-10

AI Summary

This work addresses the prevalent challenge of inter-client modality heterogeneity and intra-sample modality missingness—termed “dual-axis modality deficiency”—in multimodal federated learning. To tackle this issue, the authors propose Flux, a novel framework that, for the first time, provides a unified modeling approach for dual-axis missingness. Flux introduces a sample-level temperature modulation mechanism applied exclusively along client-private pathways, integrated with mask-aware unimodal supervision, confidence-based fusion, and gradient decoupling strategies. This design enables sample-adaptive prediction while preserving global model training stability. Extensive experiments demonstrate that Flux achieves state-of-the-art performance across four multimodal datasets, yielding an average macro-F1 improvement of 0.8–2.2 percentage points (mean +1.6) over the strongest baseline, alongside superior calibration and optimization stability.

Abstract

Multimodal federated learning (FL) supports collaborative modeling in privacy-sensitive health-sensing and medical settings, but realistic deployments often exhibit dual-axis modality missingness: clients have different modality sets, and individual samples may contain only subsets of the modalities available locally. Existing methods typically address these two axes separately. We propose Flux, a multimodal federated learning framework built around two complementary components. First, modality-aware confidence tempering learns sample-specific confidence for each modality through mask-aware unimodal supervision and fuses the confidence estimates from observed modalities into a sample-adaptive temperature that adjusts predictive sharpness according to evidence quality and completeness. Second, gradient-decoupled private adaptation applies this temperature only to a client-private prediction pathway, while training the shared federated model with a standard, untempered objective. This enables sample-specific, client-local confidence adaptation without allowing confidence-dependent gradients to perturb shared representation learning. Across four multimodal datasets, Flux achieves the highest average macro-F1 on every dataset, outperforming the strongest dataset-specific baseline by 0.8~2.2 points and by 1.6 points on average. Additional analyses demonstrate favorable calibration, temperature sensitivity to both modality missingness and input corruption, and more stable shared optimization under private-only tempering. Our code is available at https://github.com/AdibaOrz/Flux.

Problem

• multimodal federated learning
• modality missingness
• dual-axis missingness
• privacy-sensitive learning
• heterogeneous modalities

Innovation

• Multimodal Federated Learning
• Dual-Axis Modality Missingness
• Confidence Tempering
• Gradient-Decoupled Adaptation
• Sample-Adaptive Temperature