Erasing Without Collateral Damage: Precise Concept Removal in Diffusion Models

2026-07-06

AI Summary

This work addresses the issue that existing training-free methods for erasing specific concepts from diffusion models often inadvertently remove semantically related non-target content. To mitigate this, the authors propose CARE, a closed-form concept erasure operator that constructs a perceptually preserved subspace in the cross-attention value space, guided by anchor representations of retained concepts. The target concept direction is then replaced with its projection onto this subspace, enabling precise erasure while preserving shared visual structures. CARE incorporates an adjustable shrinkage parameter to balance erasure efficacy and semantic retention, and it offers theoretical guarantees of minimal perturbation. Experiments demonstrate that CARE significantly outperforms current state-of-the-art methods across instance-, style-, and celebrity-level concept erasure tasks, while effectively safeguarding unrelated semantic information.

Abstract

Training-free concept erasure is an attractive mechanism for controlling text-to-image diffusion models, but precise erasure often comes at the cost of damaging semantically related non-target concepts. Existing value-space methods remove the component of each cross-attention value along the target concept direction, implicitly treating target identity and shared visual structure as the same signal. We argue that this is the source of much of the collateral damage in prior preservation. We introduce CARE, a closed-form concept erasure operator that replaces the raw target direction with a kept-subspace-aware direction computed from a small bank of retained concept anchors. The resulting edit is applied directly in cross-attention value space, requires no model fine-tuning, and adds only a negligible offline computation. A single shrinkage parameter controls the erase-preserve trade-off. We further show that the operator admits a minimum-disturbance interpretation and, in its projection form, leaves the kept subspace invariant. Experiments under the standard concept-erasure protocol show that our method preserves non-target concepts more faithfully while maintaining competitive erasure across instance, style, and celebrity concepts. Code: https://github.com/parthupman/care

Problem

• concept erasure
• diffusion models
• collateral damage
• cross-attention
• semantic preservation

Innovation

• concept erasure
• diffusion models
• cross-attention
• subspace preservation
• training-free editing