UnGuide: Learning to Forget with LoRA-Guided Diffusion Models

2025-08-07

AI Summary

This paper addresses the challenge of *controllable concept forgetting* in diffusion models—i.e., precisely excising specific harmful or sensitive concepts without degrading overall generation fidelity. We propose **UnGuide**, a novel framework that integrates LoRA-based low-rank adaptation with a *dynamic anti-guidance mechanism*. Leveraging the stability of early denoising steps, UnGuide adaptively modulates the Classifier-Free Guidance (CFG) scale to enable fine-grained control over LoRA module activation. Its key innovation is the *first dynamic scheduling of guidance scale* to support selective concept forgetting—thereby preventing distortions in unrelated content and preserving image fidelity. Experiments demonstrate that UnGuide significantly outperforms existing LoRA-based baselines on object erasure and sensitive content removal tasks, achieving superior trade-offs between concept elimination accuracy and generative quality.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advances in large-scale text-to-image diffusion models have heightened concerns about their potential misuse, especially in generating harmful or misleading content. This underscores the urgent need for effective machine unlearning, i.e., removing specific knowledge or concepts from pretrained models without compromising overall performance. One possible approach is Low-Rank Adaptation (LoRA), which offers an efficient means to fine-tune models for targeted unlearning. However, LoRA often inadvertently alters unrelated content, leading to diminished image fidelity and realism. To address this limitation, we introduce UnGuide -- a novel approach which incorporates UnGuidance, a dynamic inference mechanism that leverages Classifier-Free Guidance (CFG) to exert precise control over the unlearning process. UnGuide modulates the guidance scale based on the stability of a few first steps of denoising processes, enabling selective unlearning by LoRA adapter. For prompts containing the erased concept, the LoRA module predominates and is counterbalanced by the base model; for unrelated prompts, the base model governs generation, preserving content fidelity. Empirical results demonstrate that UnGuide achieves controlled concept removal and retains the expressive power of diffusion models, outperforming existing LoRA-based methods in both object erasure and explicit content removal tasks.

Problem

• Removing harmful concepts from diffusion models
• Preventing unintended content alteration during unlearning
• Maintaining image fidelity while erasing specific knowledge

Innovation

• Dynamic inference mechanism for unlearning
• LoRA-guided selective concept removal
• Classifier-Free Guidance integration