Decoupled Distillation to Erase: A General Unlearning Method for Any Class-centric Tasks

2025-03-31

AI Summary

Existing class-centered unlearning methods implicitly optimize the forgetting objective while neglecting explicit supervision for retained classes, leading to pretraining distribution distortion and insufficient knowledge preservation. This paper proposes DELETE, a general unlearning framework that, for the first time, establishes a theoretically grounded decoupling of forgetting and retention objectives. DELETE achieves complete removal of target-class knowledge while precisely preserving knowledge of remaining classes via masked logit separation and dark-knowledge-driven masked distillation—without requiring access to original training data or architectural modifications to the pretrained model. Theoretical analysis confirms its soundness, and extensive experiments across face recognition, backdoor defense, and semantic segmentation demonstrate state-of-the-art performance, significantly improving the joint optimization of forgetting strength and retention fidelity.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Learning to Search
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

In this work, we present DEcoupLEd Distillation To Erase (DELETE), a general and strong unlearning method for any class-centric tasks. To derive this, we first propose a theoretical framework to analyze the general form of unlearning loss and decompose it into forgetting and retention terms. Through the theoretical framework, we point out that a class of previous methods could be mainly formulated as a loss that implicitly optimizes the forgetting term while lacking supervision for the retention term, disturbing the distribution of pre-trained model and struggling to adequately preserve knowledge of the remaining classes. To address it, we refine the retention term using"dark knowledge"and propose a mask distillation unlearning method. By applying a mask to separate forgetting logits from retention logits, our approach optimizes both the forgetting and refined retention components simultaneously, retaining knowledge of the remaining classes while ensuring thorough forgetting of the target class. Without access to the remaining data or intervention (i.e., used in some works), we achieve state-of-the-art performance across various benchmarks. What's more, DELETE is a general solution that can be applied to various downstream tasks, including face recognition, backdoor defense, and semantic segmentation with great performance.

Problem

• Proposes a general unlearning method for class-centric tasks
• Addresses inadequate knowledge retention in previous unlearning methods
• Ensures thorough forgetting while preserving remaining class knowledge

Innovation

• Decoupled distillation for class unlearning
• Mask distillation separates forgetting and retention
• Dark knowledge refines retention term