dFLMoE: Decentralized Federated Learning via Mixture of Experts for Medical Data Analysis

2025-03-13

AI Summary

To address knowledge degradation, single-point server dependency, and poor adaptability to heterogeneous data in centralized medical federated learning, this paper proposes the first decentralized Mixture of Experts (MoE) framework. It eliminates the central server by adopting a topology-aware peer-to-peer communication architecture; clients exchange only lightweight head models and locally construct personalized MoE ensembles for multi-expert decision-making. The framework integrates federated knowledge distillation with an adaptive aggregation strategy. This design substantially mitigates knowledge dilution while enhancing model robustness and generalization. Extensive experiments across diverse medical tasks demonstrate consistent superiority over state-of-the-art methods: average accuracy improves by 4.7%, communication overhead decreases by 32%, fault recovery is achieved within milliseconds, and the framework supports both homogeneous and heterogeneous client model configurations.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Multiagent Systems: Multiagent Learning

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Decentralized Web and Fediverse systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Federated learning has wide applications in the medical field. It enables knowledge sharing among different healthcare institutes while protecting patients' privacy. However, existing federated learning systems are typically centralized, requiring clients to upload client-specific knowledge to a central server for aggregation. This centralized approach would integrate the knowledge from each client into a centralized server, and the knowledge would be already undermined during the centralized integration before it reaches back to each client. Besides, the centralized approach also creates a dependency on the central server, which may affect training stability if the server malfunctions or connections are unstable. To address these issues, we propose a decentralized federated learning framework named dFLMoE. In our framework, clients directly exchange lightweight head models with each other. After exchanging, each client treats both local and received head models as individual experts, and utilizes a client-specific Mixture of Experts (MoE) approach to make collective decisions. This design not only reduces the knowledge damage with client-specific aggregations but also removes the dependency on the central server to enhance the robustness of the framework. We validate our framework on multiple medical tasks, demonstrating that our method evidently outperforms state-of-the-art approaches under both model homogeneity and heterogeneity settings.

Problem

• Decentralized federated learning for medical data analysis
• Reduces knowledge damage in client-specific aggregations
• Enhances robustness by removing central server dependency

Innovation

• Decentralized federated learning framework
• Client-specific Mixture of Experts (MoE)
• Direct lightweight head model exchange