Certified Data Removal Under High-dimensional Settings

2025-05-12

AI Summary

To address the challenge of data unlearning in high-dimensional machine learning—where full retraining must be avoided—this paper establishes, for the first time, a provably sound unlearning theory under the asymptotic regime $n/p o ext{const}$. We propose a theoretically grounded two-step Newton update algorithm, augmented by isotropic Laplace noise injection and a carefully designed scaling mechanism, which rigorously eliminates the influence of the to-be-forgotten sample. We prove that a single Newton step is insufficient for certified unlearning, whereas two steps suffice—surpassing fundamental limitations of conventional low-dimensional approaches. Numerical experiments confirm that the method strictly satisfies formal unlearning guarantees while preserving state-of-the-art predictive accuracy.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Learning to Search
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Machine unlearning focuses on the computationally efficient removal of specific training data from trained models, ensuring that the influence of forgotten data is effectively eliminated without the need for full retraining. Despite advances in low-dimensional settings, where the number of parameters ( p ) is much smaller than the sample size ( n ), extending similar theoretical guarantees to high-dimensional regimes remains challenging. We propose an unlearning algorithm that starts from the original model parameters and performs a theory-guided sequence of Newton steps ( T in { 1,2}). After this update, carefully scaled isotropic Laplacian noise is added to the estimate to ensure that any (potential) residual influence of forget data is completely removed. We show that when both ( n, p o infty ) with a fixed ratio ( n/p ), significant theoretical and computational obstacles arise due to the interplay between the complexity of the model and the finite signal-to-noise ratio. Finally, we show that, unlike in low-dimensional settings, a single Newton step is insufficient for effective unlearning in high-dimensional problems -- however, two steps are enough to achieve the desired certifiebility. We provide numerical experiments to support the certifiability and accuracy claims of this approach.

Problem

• Efficiently removing specific training data from high-dimensional models
• Extending theoretical guarantees to high-dimensional unlearning scenarios
• Ensuring complete removal of residual data influence with noise

Innovation

• Uses theory-guided Newton steps for unlearning
• Adds scaled Laplacian noise for data removal
• Requires two Newton steps for high dimensions