Efficient Verified Machine Unlearning For Distillation

2025-03-28

AI Summary

To address the high computational cost of full student model retraining when teacher models must comply with data deletion requests (e.g., under GDPR) in knowledge distillation, this paper proposes PURGE. Our method introduces three key innovations: (i) a novel component-mapping mechanism that decomposes teacher outputs into data-dependent components, each independently removable; (ii) an incremental multi-teacher collaborative distillation strategy enabling partitioned isolation and verifiable unlearning of teacher influence; and (iii) integration of SISA-style data sharding with localized student parameter updates, permitting updates only to affected submodules. We theoretically prove that PURGE significantly reduces unlearning complexity. Experiments demonstrate that, while maintaining accuracy comparable to standard distillation, PURGE achieves targeted forgetting of specified teacher-side training data by updating only a small fraction of student parameters.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Data Compression

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Growing data privacy demands, driven by regulations like GDPR and CCPA, require machine unlearning methods capable of swiftly removing the influence of specific training points. Although verified approaches like SISA, using data slicing and checkpointing, achieve efficient unlearning for single models by reverting to intermediate states, these methods struggle in teacher-student knowledge distillation settings. Unlearning in the teacher typically forces costly, complete student retraining due to pervasive information propagation during distillation. Our primary contribution is PURGE (Partitioned Unlearning with Retraining Guarantee for Ensembles), a novel framework integrating verified unlearning with distillation. We introduce constituent mapping and an incremental multi-teacher strategy that partitions the distillation process, confines each teacher constituent's impact to distinct student data subsets, and crucially maintains data isolation. The PURGE framework substantially reduces retraining overhead, requiring only partial student updates when teacher-side unlearning occurs. We provide both theoretical analysis, quantifying significant speed-ups in the unlearning process, and empirical validation on multiple datasets, demonstrating that PURGE achieves these efficiency gains while maintaining student accuracy comparable to standard baselines.

Problem

• Efficient machine unlearning for GDPR/CCPA compliance
• Verified unlearning in teacher-student distillation settings
• Reducing retraining costs during teacher-side unlearning

Innovation

• PURGE integrates verified unlearning with distillation
• Uses constituent mapping and multi-teacher strategy
• Reduces retraining overhead via data isolation