Continual Unlearning for Text-to-Image Diffusion Models: A Regularization Perspective

2025-11-11

AI Summary

This work addresses the rapid utility collapse of text-to-image diffusion models under continual unlearning—i.e., sequential processing of multiple forgetting requests—caused by parameter drift. We propose a semantic-aware gradient projection regularization method that projects parameter update directions onto the orthogonal complement of the gradient subspace spanned by retained tasks, coupled with a pre-trained weight preservation mechanism to effectively suppress cumulative parameter deviation. As the first systematic study of continual unlearning for diffusion models, our approach is compatible with existing unlearning algorithms and significantly improves post-unlearning image generation quality and semantic fidelity across multiple forgetting rounds. Quantitative evaluation shows consistent superiority over baselines in FID, CLIP Score, and human assessments. Our work establishes a novel paradigm for secure maintenance and auditable updating of generative models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Machine unlearning--the ability to remove designated concepts from a pre-trained model--has advanced rapidly, particularly for text-to-image diffusion models. However, existing methods typically assume that unlearning requests arrive all at once, whereas in practice they often arrive sequentially. We present the first systematic study of continual unlearning in text-to-image diffusion models and show that popular unlearning methods suffer from rapid utility collapse: after only a few requests, models forget retained knowledge and generate degraded images. We trace this failure to cumulative parameter drift from the pre-training weights and argue that regularization is crucial to addressing it. To this end, we study a suite of add-on regularizers that (1) mitigate drift and (2) remain compatible with existing unlearning methods. Beyond generic regularizers, we show that semantic awareness is essential for preserving concepts close to the unlearning target, and propose a gradient-projection method that constrains parameter drift orthogonal to their subspace. This substantially improves continual unlearning performance and is complementary to other regularizers for further gains. Taken together, our study establishes continual unlearning as a fundamental challenge in text-to-image generation and provides insights, baselines, and open directions for advancing safe and accountable generative AI.

Problem

• Addressing sequential unlearning requests in text-to-image diffusion models
• Mitigating cumulative parameter drift from pre-training weights via regularization
• Preserving retained knowledge while removing designated concepts continually

Innovation

• Regularization mitigates cumulative parameter drift
• Gradient-projection constrains drift orthogonal to subspace
• Semantic awareness preserves concepts near unlearning targets