Problem
Research questions and friction points this paper is trying to address.
federated learning
heterogeneous architectures
non-IID data
knowledge distillation
CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learning
🤖 AI Summary
该研究提出了一种针对异构联邦学习的去中心化知识蒸馏框架CRAD，通过评估各客户端模型并根据类别可靠性加权平均来解决非同质架构和非独立同分布数据的问题。
📝 Abstract
Conventional federated learning (FL) relies on parameter averaging, which forces clients to be doubly homogeneous: it demands an identical architecture and degrades under non-IID data. Real-world deployments usually break both assumptions. We sidestep both by building a decentralized knowledge distillation framework in which each client evaluates its peers' model snapshots on its own local data and distills from the resulting soft predictions. Because knowledge is transferred through the shared class posterior, clients are free to run different architectures; and because every teacher is evaluated on the student's own device, raw data never leaves the client, with no central server or public dataset required. Within this setting, we identify and address an under-examined problem: how to combine the peer teacher predictions. Existing methods, like uniform averaging, ignore how knowledge reliability varies across teachers and classes. We propose Class-wise Reliability-Aware Distillation (CRAD), which, per class, first discards teachers that disagree with the peer consensus and then takes a weighted average of the rest, weighting each teacher by its per-class reliability (precision, or inverse variance). Since the variance of an accuracy from $n$ samples scales as $1/n$, support enters automatically: among the teachers that survive filtering, a teacher is trusted for a class to the degree that it is both accurate and well-evidenced for it. On three image-classification benchmarks (CIFAR-10, CIFAR-100, and PathMNIST colon pathology), across heterogeneous architectures under severe non-IID skew, CRAD consistently outperforms competing methods in global accuracy.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Decentralized Knowledge Distillation
Heterogeneous Federated Learning
Class-wise Reliability-Aware Distillation
Non-IID Data
🔎 Similar Papers
No similar papers found.
