Fine-Grained Erasure in Text-to-Image Diffusion-based Foundation Models

2025-03-25

AI Summary

Existing concept forgetting methods for text-to-image diffusion models often overlook semantic proximity, leading to collateral damage to semantically related concepts during target concept erasure—the adjacency challenge. This paper introduces FADE, the first adjacency-aware fine-grained forgetting framework. FADE constructs concept neighborhoods via a semantic graph and jointly optimizes three objectives—Expungement, Adjacency preservation, and Guidance fidelity—using its novel Mesh module. It employs gradient-constrained optimization over diffusion model parameters and a multi-objective weighted loss design. Evaluated on six benchmarks including Stanford Dogs, FADE significantly reduces adjacent-concept degradation, improves knowledge retention by ≥12% over state-of-the-art methods, and maintains both thorough concept erasure and model generalization stability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Existing unlearning algorithms in text-to-image generative models often fail to preserve the knowledge of semantically related concepts when removing specific target concepts: a challenge known as adjacency. To address this, we propose FADE (Fine grained Attenuation for Diffusion Erasure), introducing adjacency aware unlearning in diffusion models. FADE comprises two components: (1) the Concept Neighborhood, which identifies an adjacency set of related concepts, and (2) Mesh Modules, employing a structured combination of Expungement, Adjacency, and Guidance loss components. These enable precise erasure of target concepts while preserving fidelity across related and unrelated concepts. Evaluated on datasets like Stanford Dogs, Oxford Flowers, CUB, I2P, Imagenette, and ImageNet1k, FADE effectively removes target concepts with minimal impact on correlated concepts, achieving atleast a 12% improvement in retention performance over state-of-the-art methods.

Problem

• Preserve related concepts during target concept removal
• Improve adjacency-aware unlearning in diffusion models
• Enhance erasure precision while maintaining concept fidelity

Innovation

• Fine grained attenuation for diffusion erasure
• Adjacency aware unlearning in diffusion models
• Mesh Modules with structured loss components