GRACE: Adaptive Concept Erasure with Geometry-Guided Retention in Diffusion Models

2026-09-11

AI Summary

为解决文本到图像扩散模型中的敏感概念删除问题，提出GRACE框架，通过几何引导的保留和自适应干预方法，在保证生成质量的同时有效移除不合规内容。

Abstract

Text-to-image (T2I) diffusion models inevitably internalize sensitive or non-compliant concepts from large-scale pretraining data, necessitating post-hoc concept erasure. However, existing erasure methods often lack explicit constraints on parameter updates, leading to over-intervention and unintended semantic drift. In addition, many methods rely on manually crafted counterfactual supervision, such as surrogate prompts, which incurs substantial data construction costs that limit scalability to new concepts. To address these limitations, we propose GRACE, a structured concept erasure framework designed to enable localized and selective intervention. Specifically, we introduce a semantically weighted sensitive subspace estimation to precisely lock intervention directions, and employ lightweight subspace-constrained adapters to prevent global semantic disturbance. To eliminate the dependency on manual prompt engineering, we design an automatically decoupled safe-anchor mechanism. To mitigate semantic drift induced by excessive intervention, we introduce an energy-driven dynamic gating mechanism that adaptively controls the timing and strength of intervention at inference. Extensive experiments demonstrate that our method achieves a superior balance between erasure effectiveness and generation fidelity. Compared with the average performance of five state-of-the-art (SOTA) concept erasure methods, our method improves the fine-grained NSFW reduction rate by $17.86\%$, while reducing the macro-averaged target CLIP Score and preservation-oriented Fréchet Inception Distance (FID) by $4.75\%$ and $50.58\%$, respectively, indicating stronger concept suppression with substantially improved preservation of the original model's generative utility.

Problem

• concept erasure
• diffusion models
• semantic drift
• parameter updates
• counterfactual supervision

Innovation

• Adaptive Concept Erasure
• Geometry-Guided Retention
• Subspace-Constrained Adapters
• Automatic Decoupled Safe-Anchor Mechanism
• Energy-Driven Dynamic Gating