MUBox: A Critical Evaluation Framework of Deep Machine Unlearning

2025-05-13

AI Summary

Existing machine unlearning methods lack a standardized, reproducible evaluation framework and multidimensional metrics for practical implementation of the “right to be forgotten.” Method: We propose MUBox—the first reproducible, unified evaluation platform tailored for real-world unlearning deployment. MUBox integrates 23 state-of-the-art unlearning algorithms, six realistic forgetting scenarios, and eleven evaluation dimensions. Contribution/Results: Our systematic evaluation reveals that current SOTA methods suffer substantial performance degradation on non-random and non-class-level forgetting tasks; relying solely on accuracy leads to severe misjudgments—robust assessment requires synergistic use of KL divergence, reverse prediction accuracy, and backdoor retention rate; and detoxification efficacy is highly attack-dependent. These findings advance unlearning research from idealized assumptions toward operationally viable, deployment-ready solutions.

Technology Category

• Machine Learning: Evaluation and Analysis
• Search and Optimization: Learning to Search
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Recent legal frameworks have mandated the right to be forgotten, obligating the removal of specific data upon user requests. Machine Unlearning has emerged as a promising solution by selectively removing learned information from machine learning models. This paper presents MUBox, a comprehensive platform designed to evaluate unlearning methods in deep learning. MUBox integrates 23 advanced unlearning techniques, tested across six practical scenarios with 11 diverse evaluation metrics. It allows researchers and practitioners to (1) assess and compare the effectiveness of different machine unlearning methods across various scenarios; (2) examine the impact of current evaluation metrics on unlearning performance; and (3) conduct detailed comparative studies on machine unlearning in a unified framework. Leveraging MUBox, we systematically evaluate these unlearning methods in deep learning and uncover several key insights: (a) Even state-of-the-art unlearning methods, including those published in top-tier venues and winners of unlearning competitions, demonstrate inconsistent effectiveness across diverse scenarios. Prior research has predominantly focused on simplified settings, such as random forgetting and class-wise unlearning, highlighting the need for broader evaluations across more difficult unlearning tasks. (b) Assessing unlearning performance remains a non-trivial problem, as no single evaluation metric can comprehensively capture the effectiveness, efficiency, and preservation of model utility. Our findings emphasize the necessity of employing multiple metrics to achieve a balanced and holistic assessment of unlearning methods. (c) In the context of depoisoning, our evaluation reveals significant variability in the effectiveness of existing approaches, which is highly dependent on the specific type of poisoning attacks.

Problem

• Evaluating effectiveness of machine unlearning methods in diverse scenarios
• Assessing impact of evaluation metrics on unlearning performance
• Comparing unlearning techniques for deep learning in unified framework

Innovation

• MUBox integrates 23 advanced unlearning techniques
• Evaluates unlearning methods with 11 diverse metrics
• Systematically tests across six practical scenarios