Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Models

2026-03-26

AI Summary

This work addresses the challenge that text-to-image diffusion models often inadvertently distort semantically adjacent concepts when erasing a specific local concept, thereby compromising fine-grained generation fidelity. To mitigate this issue, the authors propose a training-free, three-stage erasure framework that introduces, for the first time, a neighbor-aware mechanism to explicitly preserve the semantic neighborhood structure while removing the target concept. The method integrates spectral-weighted embedding modulation, attention-guided spatial gating, and a hard erasure strategy to effectively maintain the semantic integrity of related concepts. Experiments demonstrate that the approach significantly outperforms baseline methods on fine-grained datasets such as Oxford Flowers and Stanford Dogs, achieving precise removal of targets—such as celebrity identities, sensitive content, or artistic styles—while better retaining discriminative details of related categories, thus exhibiting strong generalization capability.

Abstract

Concept erasure in text-to-image diffusion models seeks to remove undesired concepts while preserving overall generative capability. Localized erasure methods aim to restrict edits to the spatial region occupied by the target concept. However, we observe that suppressing a concept can unintentionally weaken semantically related neighbor concepts, reducing fidelity in fine-grained domains. We propose Neighbor-Aware Localized Concept Erasure (NLCE), a training-free framework designed to better preserve neighboring concepts while removing target concepts. It operates in three stages: (1) a spectrally-weighted embedding modulation that attenuates target concept directions while stabilizing neighbor concept representations, (2) an attention-guided spatial gate that identifies regions exhibiting residual concept activation, and (3) a spatially-gated hard erasure that eliminates remaining traces only where necessary. This neighbor-aware pipeline enables localized concept removal while maintaining the surrounding concept neighborhood structure. Experiments on fine-grained datasets (Oxford Flowers, Stanford Dogs) show that our method effectively removes target concepts while better preserving closely related categories. Additional results on celebrity identity, explicit content and artistic style demonstrate robustness and generalization to broader erasure scenarios.

Problem

• concept erasure
• text-to-image diffusion models
• localized editing
• neighbor concepts
• fidelity preservation

Innovation

• concept erasure
• diffusion models
• localized editing
• neighbor-aware
• training-free