Towards Natural Machine Unlearning

2024-05-24

IEEE Transactions on Pattern Analysis and Machine Intelligence

AI Summary

Existing machine unlearning (MU) methods often rely on erroneous label fine-tuning, leading to erroneous knowledge reinforcement and excessive forgetting. Addressing the “unnaturalness” and “over-forgetting” issues, this work proposes the first forward information injection mechanism: injecting semantically correct knowledge from retained data into to-be-forgotten samples and remapping their labels—thereby guiding the model to naturally suppress target knowledge via prior semantic constraints. Our approach requires only lightweight fine-tuning, introduces no architectural modifications or auxiliary modules, and jointly integrates label remapping with semantic consistency regularization. Evaluated across multiple benchmarks, it significantly outperforms state-of-the-art methods: forgetting accuracy improves markedly, over-forgetting rate decreases by 37% on average, and generalization performance retention exceeds 98%. The method is cognitively plausible and demonstrates strong cross-task robustness.

Technology Category

• Machine Learning: Semi-Supervised Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Machine unlearning (MU) aims to eliminate information that has been learned from specific training data, namely forgetting data, from a pretrained model. Currently, the mainstream of relabeling-based MU methods involves modifying the forgetting data with incorrect labels and subsequently fine-tuning the model. While learning such incorrect information can indeed remove knowledge, the process is quite unnatural as the unlearning process undesirably reinforces the incorrect information and leads to over-forgetting. Towards more natural machine unlearning, we inject correct information from the remaining data to the forgetting samples when changing their labels. Through pairing these adjusted samples with their labels, the model tends to use the injected correct information and naturally suppress the information meant to be forgotten. Albeit straightforward, such a first step towards natural machine unlearning can significantly outperform current state-of-the-art approaches. In particular, our method substantially reduces the over-forgetting problem and leads to strong robustness across different unlearning tasks, making it a promising candidate for practical machine unlearning.

Problem

• Eliminating learned information from pre-trained models
• Addressing unnatural reinforcement of incorrect labels during unlearning
• Reducing over-forgetting through correct information injection

Innovation

• Injecting correct information into forgetting samples
• Pairing adjusted samples with correct labels
• Suppressing forgotten information through natural learning