WIN-U: Woodbury-Informed Newton-Unlearning as a retain-free Machine Unlearning Framework

2026-04-14

AI Summary

This work proposes an efficient machine unlearning method that satisfies the regulatory requirements of the "right to be forgotten" without accessing any retained data. Leveraging only second-order information from the original fully trained model, the approach achieves high-fidelity unlearning through a single Newton-like update, marking the first fully data-independent unlearning technique that closely approximates the gold-standard retraining. The key innovation lies in the integration of the Woodbury matrix identity with a generalized Gauss–Newton approximation to accurately emulate the effect of retraining from scratch. Experimental results demonstrate state-of-the-art unlearning performance across multiple vision and language benchmarks, maintaining model utility while exhibiting enhanced robustness against relearning attacks.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Learning to Search
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Privacy concerns in LLMs have led to the rapidly growing need to enforce a data's "right to be forgotten". Machine unlearning addresses precisely this task, namely the removal of the influence of some specific data, i.e., the forget set, from a trained model. The gold standard for unlearning is to produce the model that would have been learned on only the rest of the training data, i.e., the retain set. Most existing unlearning methods rely on direct access to the retained data, which may not be practical due to privacy or cost constraints. We propose WIN-U, a retained-data free unlearning framework that requires only second order information for the originally trained model on the full data. The unlearning is performed using a single Newton-style step. Using the Woodbury matrix identity and a generalized Gauss-Newton approximation for the forget set curvature, the WIN-U update recovers the closed-form linear solution and serves as a local second-order approximation to the gold-standard retraining optimum. Extensive experiments on various vision and language benchmarks demonstrate that WIN-U achieves SOTA performance in terms of unlearning efficacy and utility preservation, while being more robust against relearning attacks compared to existing methods. Importantly, WIN-U does not require access to the retained data.

Problem

• machine unlearning
• right to be forgotten
• retain-free
• privacy
• large language models

Innovation

• machine unlearning
• retain-free
• Woodbury identity
• Newton method
• second-order approximation