SFedKD: Sequential Federated Learning with Discrepancy-Aware Multi-Teacher Knowledge Distillation

2025-07-11

AI Summary

To address severe catastrophic forgetting in sequential federated learning (SFL) under data heterogeneity, this paper proposes a Difference-Aware Multi-Teacher Knowledge Distillation (DA-MTKD) framework. DA-MTKD introduces a fine-grained, class-distribution-difference-based weighting mechanism to dynamically prioritize knowledge contributions from target versus non-target classes. It formulates teacher selection as a maximum coverage problem and devises a complementarity-driven greedy strategy to prevent knowledge dilution. Furthermore, it extends knowledge distillation in a decoupled manner to enable efficient fusion of heterogeneous knowledge from multiple teachers. Extensive experiments on diverse heterogeneous benchmarks demonstrate that DA-MTKD significantly outperforms existing SFL methods, effectively mitigating forgetting while enhancing global model generalization, stability, and convergence speed.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Federated Learning (FL) is a distributed machine learning paradigm which coordinates multiple clients to collaboratively train a global model via a central server. Sequential Federated Learning (SFL) is a newly-emerging FL training framework where the global model is trained in a sequential manner across clients. Since SFL can provide strong convergence guarantees under data heterogeneity, it has attracted significant research attention in recent years. However, experiments show that SFL suffers from severe catastrophic forgetting in heterogeneous environments, meaning that the model tends to forget knowledge learned from previous clients. To address this issue, we propose an SFL framework with discrepancy-aware multi-teacher knowledge distillation, called SFedKD, which selects multiple models from the previous round to guide the current round of training. In SFedKD, we extend the single-teacher Decoupled Knowledge Distillation approach to our multi-teacher setting and assign distinct weights to teachers' target-class and non-target-class knowledge based on the class distributional discrepancy between teacher and student data. Through this fine-grained weighting strategy, SFedKD can enhance model training efficacy while mitigating catastrophic forgetting. Additionally, to prevent knowledge dilution, we eliminate redundant teachers for the knowledge distillation and formalize it as a variant of the maximum coverage problem. Based on the greedy strategy, we design a complementary-based teacher selection mechanism to ensure that the selected teachers achieve comprehensive knowledge space coverage while reducing communication and computational costs. Extensive experiments show that SFedKD effectively overcomes catastrophic forgetting in SFL and outperforms state-of-the-art FL methods.

Problem

• Mitigates catastrophic forgetting in Sequential Federated Learning
• Enhances training via multi-teacher knowledge distillation
• Reduces redundancy in teacher models for efficiency

Innovation

• Multi-teacher knowledge distillation for SFL
• Discrepancy-aware weighting for class knowledge
• Greedy-based teacher selection to reduce redundancy