Variance-Reduced $(\varepsilon,\delta)-$Unlearning using Forget Set Gradients

2026-02-16

AI Summary

This work addresses the challenge of effectively leveraging the forget set as an optimization signal in machine unlearning while satisfying formal $(\varepsilon,\delta)$-unlearning guarantees. We propose Variance-Reduced Unlearning (VRU), the first algorithm that directly incorporates gradients from the forget set into the update rule within a first-order $(\varepsilon,\delta)$-unlearning framework, combining differential privacy noise with variance reduction techniques. Theoretical analysis demonstrates that VRU achieves a superior error-dependent convergence rate under strong convexity and strictly outperforms existing methods that ignore the forget set in low-error regimes. Empirical evaluations confirm consistent improvements in both certified unlearning performance and practical utility.

Technology Category

• Machine Learning: Privacy
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

In machine unlearning, $(\varepsilon,\delta)-$unlearning is a popular framework that provides formal guarantees on the effectiveness of the removal of a subset of training data, the forget set, from a trained model. For strongly convex objectives, existing first-order methods achieve $(\varepsilon,\delta)-$unlearning, but they only use the forget set to calibrate injected noise, never as a direct optimization signal. In contrast, efficient empirical heuristics often exploit the forget samples (e.g., via gradient ascent) but come with no formal unlearning guarantees. We bridge this gap by presenting the Variance-Reduced Unlearning (VRU) algorithm. To the best of our knowledge, VRU is the first first-order algorithm that directly includes forget set gradients in its update rule, while provably satisfying ($(\varepsilon,\delta)-$unlearning. We establish the convergence of VRU and show that incorporating the forget set yields strictly improved rates, i.e. a better dependence on the achieved error compared to existing first-order $(\varepsilon,\delta)-$unlearning methods. Moreover, we prove that, in a low-error regime, VRU asymptotically outperforms any first-order method that ignores the forget set.Experiments corroborate our theory, showing consistent gains over both state-of-the-art certified unlearning methods and over empirical baselines that explicitly leverage the forget set.

Problem

• machine unlearning
• variance reduction
• forget set gradients
• differential privacy
• first-order methods

Innovation

• variance-reduced unlearning
• forget set gradients
• first-order algorithm
• convergence rate
• (ε,δ)-unlearning