On the Byzantine-Resilience of Distillation-Based Federated Learning

2024-02-19

arXiv.org

AI Summary

This work investigates the Byzantine robustness of knowledge distillation-based federated learning (KD-FL). Addressing the lack of systematic analysis of KD-FL’s intrinsic interference resilience, we: (1) propose two novel stealthy Byzantine attacks and establish a unified attack confusion framework, significantly increasing attack success rates and evading detection; (2) design an enhanced robust aggregation mechanism integrating public-dataset-based prediction sharing and gradient calibration; and (3) theoretically and empirically demonstrate KD-FL’s inherent robustness, showing our defense improves model accuracy by up to 32% under diverse Byzantine attacks—substantially outperforming state-of-the-art Byzantine-resilient aggregation methods. Our results reveal that knowledge distillation not only enhances communication efficiency but also intrinsically strengthens system resilience against adversarial clients.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Federated Learning (FL) algorithms using Knowledge Distillation (KD) have received increasing attention due to their favorable properties with respect to privacy, non-i.i.d. data and communication cost. These methods depart from transmitting model parameters and instead communicate information about a learning task by sharing predictions on a public dataset. In this work, we study the performance of such approaches in the byzantine setting, where a subset of the clients act in an adversarial manner aiming to disrupt the learning process. We show that KD-based FL algorithms are remarkably resilient and analyze how byzantine clients can influence the learning process. Based on these insights, we introduce two new byzantine attacks and demonstrate their ability to break existing byzantine-resilient methods. Additionally, we propose a novel defence method which enhances the byzantine resilience of KD-based FL algorithms. Finally, we provide a general framework to obfuscate attacks, making them significantly harder to detect, thereby improving their effectiveness. Our findings serve as an important building block in the analysis of byzantine FL, contributing through the development of new attacks and new defence mechanisms, further advancing the robustness of KD-based FL algorithms.

Problem

• Analyze Byzantine resilience in Knowledge Distillation-based Federated Learning.
• Introduce new Byzantine attacks to disrupt existing resilient methods.
• Propose a defense method to enhance Byzantine resilience in FL.

Innovation

• KD-based FL algorithms resist Byzantine attacks
• Introduces new Byzantine attacks and defense methods
• Framework to obfuscate attacks, enhancing their effectiveness