Robust Decentralized Federated Distillation via Multi-Modality Knowledge Collaboration

2026-09-07

AI Summary

本文提出了一种鲁棒的去中心化联邦蒸馏方法，通过多模态知识协作解决异构模型间在非IID数据和拜占庭攻击下的预测准确性问题。

Abstract

This paper propose a robust decentralized federated distillation method that enables clients with heterogeneous models to collaborate through predictions on shared unlabeled public data. In the proposed method, each client first evaluates the received predictions in three modalities of class prediction, boundary decision, and prediction correlation. It then filters unreliable clients, assigns reliability-based weights to the retained clients, and constructs a teacher for each type of knowledge. Finally, the corresponding distillation gradients are validated using a supervised gradient computed from private data. Conflicting prediction and boundary gradients are removed, and conflicting relation gradients are suppressed before the final model update. We prove the convergence of the proposed method by showing stable local optimization for honest clients under Byzantine distillation. Particularly, we show that our method ensures a bounded Byzantine influence on both distillation gradients and individual client private gradients after cross-modality fusion, thereby enabling stable local optimization for honest clienunder Byzantine distillation. Extensive experiments on CIFAR-10 and CIFAR-100 demonstrate that the proposed method improves the prediction accuracy of heterogeneous models of clients under non-IID data and Byzantine attacks. As the booming demands of federated learning in decentralized environments such as edge computing and mission-oriented UAV collaborations, our method has a great potential for adoption of DFL in unreliable real-world scenarios where clients are exposed to receiver-specific Byzantine messages of malicious predictions.

Problem

• Decentralized Federated Distillation
• Heterogeneous Models
• Byzantine Attacks
• Non-IID Data

Innovation

• decentralized federated distillation
• multi-modality knowledge collaboration
• Byzantine attacks resistance
• heterogeneous models
• non-IID data