Federated Distillation on Edge Devices: Efficient Client-Side Filtering for Non-IID Data

2025-08-20

AI Summary

To address the high computational overhead of density ratio estimation and server-side knowledge filtering latency in federated distillation under non-IID data, this paper proposes EdgeFD—a lightweight federated distillation framework tailored for edge devices. Its core innovation is a KMeans-based lightweight density ratio estimator deployed locally on clients, enabling efficient in-distribution vs. out-of-distribution proxy data discrimination and selective upload of soft logits—thereby eliminating the need for server-side ambiguous knowledge filtering. EdgeFD requires no pre-trained teacher model and enables low-overhead knowledge transfer. Experiments demonstrate that EdgeFD achieves accuracy close to the ideal IID baseline across strong/weak non-IID and IID settings, while significantly reducing both computational and communication costs. This substantially improves scalability and practicality of federated distillation on resource-constrained edge devices.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Federated distillation has emerged as a promising collaborative machine learning approach, offering enhanced privacy protection and reduced communication compared to traditional federated learning by exchanging model outputs (soft logits) rather than full model parameters. However, existing methods employ complex selective knowledge-sharing strategies that require clients to identify in-distribution proxy data through computationally expensive statistical density ratio estimators. Additionally, server-side filtering of ambiguous knowledge introduces latency to the process. To address these challenges, we propose a robust, resource-efficient EdgeFD method that reduces the complexity of the client-side density ratio estimation and removes the need for server-side filtering. EdgeFD introduces an efficient KMeans-based density ratio estimator for effectively filtering both in-distribution and out-of-distribution proxy data on clients, significantly improving the quality of knowledge sharing. We evaluate EdgeFD across diverse practical scenarios, including strong non-IID, weak non-IID, and IID data distributions on clients, without requiring a pre-trained teacher model on the server for knowledge distillation. Experimental results demonstrate that EdgeFD outperforms state-of-the-art methods, consistently achieving accuracy levels close to IID scenarios even under heterogeneous and challenging conditions. The significantly reduced computational overhead of the KMeans-based estimator is suitable for deployment on resource-constrained edge devices, thereby enhancing the scalability and real-world applicability of federated distillation. The code is available online for reproducibility.

Problem

• Efficient client-side filtering for non-IID data in federated distillation
• Reducing computational complexity of density ratio estimation on edge devices
• Eliminating server-side filtering latency while maintaining accuracy under heterogeneity

Innovation

• KMeans-based efficient density ratio estimator
• Client-side filtering for non-IID data
• Eliminates server-side filtering latency