EraseAnything: Enabling Concept Erasure in Rectified Flow Transformers

2024-12-29

AI Summary

Emerging flow-matching text-to-image (T2I) models—such as Stable Diffusion v3 and Flux—exhibit limited controllability in precisely erasing sensitive concepts (e.g., brands, individuals) without degrading generation quality. Method: This work introduces the first concept-erasure framework tailored to flow-matching + Transformer architectures. It proposes a two-level optimization framework integrating attention-map regularization and dual-stage self-contrastive learning to jointly ensure erasure specificity and generalization robustness; further enhanced by LoRA fine-tuning, attention-visualization constraints, and flow-matching gradient alignment. Results: The method achieves state-of-the-art performance across diverse sensitive-concept erasure tasks: 37% higher erasure success rate, 98.2% fidelity preservation for unrelated concepts, and no statistically significant degradation in generation quality. Core contribution: The first controllable, concept-level erasure paradigm for flow-matching T2I models, underpinned by an efficient, synergistic optimization mechanism.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Removing unwanted concepts from large-scale text-to-image (T2I) diffusion models while maintaining their overall generative quality remains an open challenge. This difficulty is especially pronounced in emerging paradigms, such as Stable Diffusion (SD) v3 and Flux, which incorporate flow matching and transformer-based architectures. These advancements limit the transferability of existing concept-erasure techniques that were originally designed for the previous T2I paradigm (e.g., SD v1.4). In this work, we introduce EraseAnything, the first method specifically developed to address concept erasure within the latest flow-based T2I framework. We formulate concept erasure as a bi-level optimization problem, employing LoRA-based parameter tuning and an attention map regularizer to selectively suppress undesirable activations. Furthermore, we propose a self-contrastive learning strategy to ensure that removing unwanted concepts does not inadvertently harm performance on unrelated ones. Experimental results demonstrate that EraseAnything successfully fills the research gap left by earlier methods in this new T2I paradigm, achieving state-of-the-art performance across a wide range of concept erasure tasks.

Problem

• Information Filtering
• Model Performance
• Unwanted Concepts Removal

Innovation

• EraseAnything
• Selective Information Ignoring
• Performance Preservation