Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality

2024-01-25

arXiv.org

AI Summary

Multimodal federated learning (MFL) suffers from severe modality missing—up to 80%—due to sensor failures, leading to feature misalignment, knowledge fragmentation, and degraded model generalization. Method: We propose a cross-modal prototype collaboration mechanism, the first framework explicitly designed for severe modality missing in MFL. It integrates dual-level contrastive learning (modal-shared and modality-specific), cross-modal prototype alignment regularization, and a robust federated training strategy combining federated averaging with zero-padding. Contribution/Results: Evaluated on three multimodal benchmarks, our method significantly outperforms state-of-the-art approaches: under 80% modality missing, it retains over 92% of baseline performance, establishing, for the first time, robustness and generalization of the global model in highly sparse modality scenarios.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multimodal federated learning (MFL) has emerged as a decentralized machine learning paradigm, allowing multiple clients with different modalities to collaborate on training a global model across diverse data sources without sharing their private data. However, challenges, such as data heterogeneity and severely missing modalities, pose crucial hindrances to the robustness of MFL, significantly impacting the performance of global model. The occurrence of missing modalities in real-world applications, such as autonomous driving, often arises from factors like sensor failures, leading knowledge gaps during the training process. Specifically, the absence of a modality introduces misalignment during the local training phase, stemming from zero-filling in the case of clients with missing modalities. Consequently, achieving robust generalization in global model becomes imperative, especially when dealing with clients that have incomplete data. In this paper, we propose $ extbf{Multimodal Federated Cross Prototype Learning (MFCPL)}$, a novel approach for MFL under severely missing modalities. Our MFCPL leverages the complete prototypes to provide diverse modality knowledge in modality-shared level with the cross-modal regularization and modality-specific level with cross-modal contrastive mechanism. Additionally, our approach introduces the cross-modal alignment to provide regularization for modality-specific features, thereby enhancing the overall performance, particularly in scenarios involving severely missing modalities. Through extensive experiments on three multimodal datasets, we demonstrate the effectiveness of MFCPL in mitigating the challenges of data heterogeneity and severely missing modalities while improving the overall performance and robustness of MFL.

Problem

• Addresses data heterogeneity in multimodal federated learning.
• Mitigates performance issues from severely missing modalities.
• Enhances global model robustness with cross-modal alignment.

Innovation

• Cross-modal regularization enhances shared modality knowledge.
• Cross-modal contrastive mechanism improves modality-specific features.
• Cross-modal alignment addresses severely missing modality challenges.