Concept Removal Guidance: Evidence-Calibrated Negative Guidance for Safe Diffusion Sampling

2026-06-29

AI Summary

This work addresses the vulnerability of text-to-image diffusion models to adversarial prompts that induce generation of prohibited content, a challenge inadequately mitigated by existing negative prompting techniques which often compromise either safety or output fidelity. The authors propose a training-free, dynamic negative guidance calibration method that, at each denoising step, estimates the presence of restricted concepts based on the model’s own noise predictions and adaptively adjusts guidance strength via closed-form constrained optimization to meet a target suppression threshold with minimal perturbation. Notably, this approach leverages image evidence from the diffusion trajectory itself for dynamic calibration—requiring no external classifiers or fine-tuning—and flexibly supports removal of diverse sensitive content, including violence and artist-style mimicry. Experiments demonstrate significantly reduced attack success rates under red-teaming evaluations while preserving high-fidelity generation for benign prompts, with strong generalization across multiple content safety tasks.

Abstract

Text-to-image diffusion models remain vulnerable to adversarial prompts that elicit disallowed content, motivating reliable inference-time controls. A popular approach is negative guidance, which subtracts a negative prompt direction with a fixed weight. However, it often forces a safety-fidelity trade-off, causing artifacts or prompt drift when over-applied and failing under attacks when under-applied. Dynamic variants reweight guidance using posterior-odds signals, which can be brittle for open-vocabulary compositional prompts, while lightweight similarity-based methods ignore the evolving image evidence along the denoising trajectory. We introduce Concept Removal Guidance (CRG), a training-free method that estimates unwanted-concept presence at each diffusion step from the model's noise predictions, and adaptively calibrates negative guidance via a closed-form constrained update enforcing a target presence threshold while minimally perturbing the conditional trajectory. Across red-teaming benchmarks, CRG reduces attack success rates while preserving benign fidelity, and extends to additional suppression targets such as artist style and violence without fine-tuning or external classifiers.

Problem

• adversarial prompts
• negative guidance
• safety-fidelity trade-off
• diffusion models
• concept removal

Innovation

• Concept Removal Guidance
• negative guidance
• diffusion models
• inference-time control
• evidence calibration