UNO: Unlearning via Orthogonalization in Generative models

2025-06-05

AI Summary

To address the challenge of efficiently unlearning specific training examples from generative models without full retraining, this paper proposes a controllable unlearning method based on gradient orthogonalization. The core innovation lies in the first application of loss-gradient orthogonalization to unlearning in generative models (VAEs and GANs), enabling strict suppression of gradient contributions from target data via only one or a few parameter updates, while preserving learning signals from all other data. Evaluated on MNIST and CelebA, the method achieves unlearning speeds one to two orders of magnitude faster than baseline approaches such as gradient surgery; it incurs less than 2% degradation in generation quality and retains over 98% of original task performance. Thus, it simultaneously attains high-fidelity unlearning, minimal generative distortion, and strong knowledge retention—establishing a principled three-way trade-off.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

As generative models become increasingly powerful and pervasive, the ability to unlearn specific data, whether due to privacy concerns, legal requirements, or the correction of harmful content, has become increasingly important. Unlike in conventional training, where data are accumulated and knowledge is reinforced, unlearning aims to selectively remove the influence of particular data points without costly retraining from scratch. To be effective and reliable, such algorithms need to achieve (i) forgetting of the undesired data, (ii) preservation of the quality of the generation, (iii) preservation of the influence of the desired training data on the model parameters, and (iv) small number of training steps. We propose fast unlearning algorithms based on loss gradient orthogonalization. We show that our algorithms are able to forget data while maintaining the fidelity of the original model. Using MNIST and CelebA data, we demonstrate that our algorithms achieve orders of magnitude faster unlearning times than their predecessors, such as gradient surgery.

Problem

• Selectively remove specific data influence in generative models
• Maintain generation quality while forgetting undesired data
• Achieve fast unlearning without costly retraining

Innovation

• Orthogonalization for generative model unlearning
• Fast unlearning via loss gradient orthogonalization
• Maintains model fidelity post-unlearning