Clustered Federated Learning with Hierarchical Knowledge Distillation

2025-12-11

AI Summary

Traditional clustered federated learning (CFL) suffers from fragmented learning and limited inter-cluster knowledge sharing due to isolated model training within clusters. To address this, this paper proposes a hierarchical clustered federated learning framework tailored for IoT environments. The framework introduces a two-tier aggregation mechanism—local clustering modeling at the edge and unified global modeling in the cloud—and pioneers a hierarchical knowledge distillation method that enables multi-teacher, cross-cluster knowledge transfer, thereby preserving cluster-level personalization while integrating collective learning insights. Experiments on standard benchmark datasets demonstrate that the proposed approach significantly outperforms mainstream baselines in both cluster-level and global model accuracy, with improvements of 3.32%–7.57%. This achieves effective co-optimization of personalization capability and generalization performance.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Humans and AI: Human-in-the-loop Machine Learning
• Multiagent Systems: Multiagent Learning

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Clustered Federated Learning (CFL) has emerged as a powerful approach for addressing data heterogeneity and ensuring privacy in large distributed IoT environments. By clustering clients and training cluster-specific models, CFL enables personalized models tailored to groups of heterogeneous clients. However, conventional CFL approaches suffer from fragmented learning for training independent global models for each cluster and fail to take advantage of collective cluster insights. This paper advocates a shift to hierarchical CFL, allowing bi-level aggregation to train cluster-specific models at the edge and a unified global model at the cloud. This shift improves training efficiency yet might introduce communication challenges. To this end, we propose CFLHKD, a novel personalization scheme for integrating hierarchical cluster knowledge into CFL. Built upon multi-teacher knowledge distillation, CFLHKD enables inter-cluster knowledge sharing while preserving cluster-specific personalization. CFLHKD adopts a bi-level aggregation to bridge the gap between local and global learning. Extensive evaluations of standard benchmark datasets demonstrate that CFLHKD outperforms representative baselines in cluster-specific and global model accuracy and achieves a performance improvement of 3.32-7.57%.

Problem

• Addresses data heterogeneity and privacy in federated learning
• Enables inter-cluster knowledge sharing while preserving personalization
• Improves training efficiency and model accuracy through hierarchical distillation

Innovation

• Hierarchical clustering with bi-level aggregation for federated learning
• Multi-teacher knowledge distillation for inter-cluster knowledge sharing
• Personalized models via cluster-specific and unified global training