Auditing Machine Unlearning: A Systematic Research on Whether Models Truly Forget

2026-06-14

AI Summary

This work addresses the critical gap in existing machine unlearning algorithms: the lack of reliable and practical auditing mechanisms to verify whether the influence of specified data has been truly removed, thereby posing privacy risks. The paper proposes the first general and practical machine unlearning audit framework, inspired by the concept of "proof of ignorance." Notably, this framework operates without requiring full retraining of baseline models, reliance on shadow models, or any modification to the original training process. Through black-box auditing and cross-dataset/cross-model generalization evaluation, it systematically assesses the real-world effectiveness of diverse mainstream unlearning methods—including retraining, fine-tuning, deoptimization, and Fisher/Hessian-based approaches. Experiments reveal that only retraining and fine-tuning achieve effective unlearning, while some formally certified methods (e.g., deoptimization-based) fail to genuinely forget. The framework demonstrates robustness against false unlearning claims and scales effectively to large language models.

Abstract

Machine unlearning has been extensively studied in response to growing privacy concerns and regulatory requirements. However, auditing whether unlearning algorithms have truly erased the influence of specific data remains an open challenge. The lack of reliable and practical auditing mechanisms can lead to critical privacy risks, such as residual information leakage. This paper initiates a systematic investigation into whether existing unlearning algorithms can truly forget the designated data. We propose the first practical and general-purpose auditing framework for machine unlearning, inspired by the concept of proof of ignorance. Our framework addresses the key practicality limitations of existing methods by eliminating the need for retraining-from-scratch baselines, avoiding the training of large numbers of shadow models, and requiring no intrusive intervention in the original training process. To evaluate the effectiveness of our framework, we first conduct validation experiments to verify its soundness and completeness. We then perform comprehensive experiments across six datasets and ten representative unlearning methods. The results demonstrate that our framework reliably distinguishes between successful and failed unlearning. In particular, we observe that retraining-based and fine-tuning-based methods can achieve effective unlearning, even when the target data remain in the original dataset. In contrast, de-optimization-based methods fail to achieve true unlearning and instead degrade the model's performance. Fisher/Hessian-based methods also fail to unlearn requested data, even formal certification is provided. Moreover, we show that our framework is robust against fake unlearning attempts and generalizes well to large language models.

Problem

• machine unlearning
• auditing
• privacy
• data forgetting
• information leakage

Innovation

• machine unlearning
• auditing framework
• proof of ignorance
• privacy verification
• large language models