HFedCKD: Toward Robust Heterogeneous Federated Learning via Data-free Knowledge Distillation and Two-way Contrast

2025-03-09

AI Summary

To address critical challenges in system-heterogeneous federated learning—including low client participation, uneven contribution, and knowledge drift—this paper proposes HFedCKD, a novel framework integrating data-free knowledge distillation with bidirectional contrastive learning. Methodologically, HFedCKD introduces (1) inverse probability weighted distillation (IPWD), a dynamic mechanism to quantify each client’s predictive contribution; (2) decoupled feature extractors and classifiers, jointly optimized via discriminative contrastive learning to align representations while preserving personalized decision boundaries; and (3) generative feature completion to mitigate performance degradation caused by local data scarcity. Extensive experiments on image and IoT benchmarks demonstrate that HFedCKD significantly improves model accuracy and robustness, effectively alleviates performance drop under low participation, and enhances fairness and stability in cross-client knowledge fusion across heterogeneous systems.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Most current federated learning frameworks are modeled as static processes, ignoring the dynamic characteristics of the learning system. Under the limited communication budget of the central server, the flexible model architecture of a large number of clients participating in knowledge transfer requires a lower participation rate, active clients have uneven contributions, and the client scale seriously hinders the performance of FL. We consider a more general and practical federation scenario and propose a system heterogeneous federation method based on data-free knowledge distillation and two-way contrast (HFedCKD). We apply the Inverse Probability Weighted Distillation (IPWD) strategy to the data-free knowledge transfer framework. The generator completes the data features of the nonparticipating clients. IPWD implements a dynamic evaluation of the prediction contribution of each client under different data distributions. Based on the antibiased weighting of its prediction loss, the weight distribution of each client is effectively adjusted to fairly integrate the knowledge of participating clients. At the same time, the local model is split into a feature extractor and a classifier. Through differential contrast learning, the feature extractor is aligned with the global model in the feature space, while the classifier maintains personalized decision-making capabilities. HFedCKD effectively alleviates the knowledge offset caused by a low participation rate under data-free knowledge distillation and improves the performance and stability of the model. We conduct extensive experiments on image and IoT datasets to comprehensively evaluate and verify the generalization and robustness of the proposed HFedCKD framework.

Problem

• Addresses dynamic characteristics in federated learning systems.
• Improves client participation and contribution fairness in FL.
• Enhances model performance and stability via data-free knowledge distillation.

Innovation

• Data-free knowledge distillation for heterogeneous federated learning
• Two-way contrast learning for feature and classifier alignment
• Inverse Probability Weighted Distillation for dynamic client evaluation