FedQUIT: On-Device Federated Unlearning via a Quasi-Competent Virtual Teacher

2024-08-14

arXiv.org

AI Summary

In federated learning, efficiently removing a user’s contribution upon exercising the “right to be forgotten” remains challenging without accessing raw data or historical model updates. This paper proposes a lightweight, client-side localized model unlearning method. Our approach constructs a teacher-student framework based on knowledge distillation and introduces a novel “pseudo-competent teacher” mechanism that achieves precise unlearning by dynamically suppressing predictions for the target class—requiring neither proxy data, historical gradients, nor centralized data access. The method is fully compatible with FedAvg, necessitating only a single round of local fine-tuning with computational overhead equivalent to one standard FedAvg update. Communication cost is reduced by up to 117.6× compared to full retraining. Empirical evaluation demonstrates that the unlearned model retains superior generalization performance over existing state-of-the-art unlearning methods.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Federated Learning (FL) systems enable the collaborative training of machine learning models without requiring centralized collection of individual data. FL participants should have the ability to exercise their right to be forgotten, ensuring their past contributions can be removed from the learned model upon request. In this paper, we propose FedQUIT, a novel algorithm that uses knowledge distillation to scrub the contribution of the data to forget from an FL global model while preserving its generalization ability. FedQUIT directly works on client devices that request to leave the federation, and leverages a teacher-student framework. The FL global model acts as the teacher, and the local model works as the student. To induce forgetting, FedQUIT tailors the teacher's output on local data (the data to forget) penalizing the prediction score of the true class. Unlike previous work, our method does not require hardly viable assumptions for cross-device settings, such as storing historical updates of participants or requiring access to proxy datasets. Experimental results on various datasets and model architectures demonstrate that (i) FedQUIT outperforms state-of-the-art competitors in forgetting data, (ii) has the exact computational requirements as a regular FedAvg round, and (iii) reduces the cumulative communication costs by up to 117.6$ imes$ compared to retraining from scratch to restore the initial generalization performance after unlearning.

Problem

• Enables federated unlearning for data removal in FL systems
• Preserves model generalization while scrubbing forgotten data contributions
• Reduces communication costs and avoids impractical assumptions

Innovation

• On-device federated unlearning via knowledge distillation
• Quasi-competent virtual teacher for forgetting data
• No historical updates or proxy datasets needed