Forgetting Similar Samples: Can Machine Unlearning Do it Better?

2026-01-11

arXiv.org

AI Summary

This study addresses a critical limitation in current machine unlearning methods: their inability to fully eliminate the influence of a target sample when highly similar instances remain in the training set. We present the first systematic evaluation of mainstream unlearning approaches—including full retraining—under such conditions, demonstrating that none consistently satisfy the fundamental requirement of completely removing the effect of specific data points. Through experiments on four carefully constructed datasets containing semantically or visually similar samples, using both vision and language models, we reveal significant performance deficiencies across existing techniques. Our findings not only expose the practical shortcomings of current unlearning strategies but also challenge prevailing assumptions about what constitutes effective unlearning, thereby prompting a reexamination of its theoretical foundations and offering clear directions for future research.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Machine unlearning, a process enabling pre-trained models to remove the influence of specific training samples, has attracted significant attention in recent years. Although extensive research has focused on developing efficient machine unlearning strategies, we argue that these methods mainly aim at removing samples rather than removing samples'influence on the model, thus overlooking the fundamental definition of machine unlearning. In this paper, we first conduct a comprehensive study to evaluate the effectiveness of existing unlearning schemes when the training dataset includes many samples similar to those targeted for unlearning. Specifically, we evaluate: Do existing unlearning methods truly adhere to the original definition of machine unlearning and effectively eliminate all influence of target samples when similar samples are present in the training dataset? Our extensive experiments, conducted on four carefully constructed datasets with thorough analysis, reveal a notable gap between the expected and actual performance of most existing unlearning methods for image and language models, even for the retraining-from-scratch baseline. Additionally, we also explore potential solutions to enhance current unlearning approaches.

Problem

• machine unlearning
• similar samples
• influence removal
• unlearning effectiveness
• model forgetting

Innovation

• machine unlearning
• influence removal
• similar samples
• unlearning evaluation
• model retraining