The Utility and Complexity of In- and Out-of-Distribution Machine Unlearning

2024-12-12

arXiv.org

AI Summary

This paper addresses machine unlearning—the efficient removal of specific data from a trained model to ensure privacy and mitigate knowledge gaps—particularly under both in-distribution (ID) and out-of-distribution (OOD) forgetting scenarios. Existing approaches lack theoretical guarantees and often require super-retraining time for single-sample OOD unlearning. The authors propose: (1) the first rigorous unlearning certification framework grounded in class-level differential privacy; (2) a proof that ID unlearning admits an optimal utility–privacy trade-off via output-perturbed empirical risk minimization; and (3) a robust noisy gradient descent algorithm that breaks the fundamental time-complexity lower bound for OOD unlearning, achieving amortized near-linear-time unlearning with provably zero utility loss. Theoretical analysis establishes tight bounds on the interplay among unlearning accuracy, model utility, and time–space complexity.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Security and Privacy: Security and privacy of machine learning and AI applications
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Machine unlearning, the process of selectively removing data from trained models, is increasingly crucial for addressing privacy concerns and knowledge gaps post-deployment. Despite this importance, existing approaches are often heuristic and lack formal guarantees. In this paper, we analyze the fundamental utility, time, and space complexity trade-offs of approximate unlearning, providing rigorous certification analogous to differential privacy. For in-distribution forget data -- data similar to the retain set -- we show that a surprisingly simple and general procedure, empirical risk minimization with output perturbation, achieves tight unlearning-utility-complexity trade-offs, addressing a previous theoretical gap on the separation from unlearning"for free"via differential privacy, which inherently facilitates the removal of such data. However, such techniques fail with out-of-distribution forget data -- data significantly different from the retain set -- where unlearning time complexity can exceed that of retraining, even for a single sample. To address this, we propose a new robust and noisy gradient descent variant that provably amortizes unlearning time complexity without compromising utility.

Problem

• Analyzes complexity and utility of machine unlearning.
• Addresses unlearning for in-distribution data efficiently.
• Solves high complexity in out-of-distribution data unlearning.

Innovation

• Machine unlearning via empirical risk minimization
• Robust noisy gradient descent for out-distribution data
• Rigorous certification akin to differential privacy