AEGIS: A Mechanism-Guided Defense against Visual Synonym Jailbreaks in Text-to-Image Models

2026-07-07

AI Summary

This work addresses the vulnerability of current text-to-image models to visual synonym attacks (VSAs), which exploit semantically implicit yet benign prompts to generate prohibited content, thereby undermining conventional defenses that struggle to balance safety and utility. The authors propose a mechanism-informed, inference-time defense framework that, for the first time, reveals how both VSA and explicitly harmful prompts converge during generation through sparse semantic injection into specific attention heads. Leveraging this insight, they design an adaptive strategy that dynamically monitors the generation process, precisely identifies vulnerable attention heads, and applies similarity-aware semantic repulsion. Evaluated on Stable Diffusion 1.4, the method reduces attack success rates to 0.00 for violence and 0.03 for nudity under VSA, achieves ≤0.09 ASR against out-of-domain and adversarial attacks, preserves fidelity of benign images without over-suppression, and successfully generalizes to SD 2.1 and FLUX.1.

Abstract

Text-to-image diffusion models have achieved high visual fidelity and broad adoption, but remain vulnerable to safety violations when adversaries exploit them to synthesize illicit content. Existing alignment paradigms, from input sanitization to structural feature pruning, are largely organized around unsafe concepts explicitly exposed during filtering, editing, or localization. This leaves a blind spot for visual synonym attacks (VSA), a jailbreak where benign-looking prompts elicit prohibited imagery through implicit visual associations. As a result, current defenses face a safety-utility dilemma: they may either under-mitigate VSA threats or over-suppress visually similar benign concepts. The core challenge is that VSA hides the unsafe target at the textual surface while revealing it through generation-time visual-semantic convergence. In this work, we therefore shift from static suppression of pre-specified unsafe concepts to dynamic tracing of how unsafe semantics emerge during generation. Our mechanistic analysis shows that VSA and explicit unsafe prompts converge through sparse semantic-injecting attention heads, which serve as inference-time bottlenecks for prohibited visual semantics. Based on this insight, we propose AEGIS (Adaptive Evasion Guard via Identification and Steering), an inference-time defense that applies similarity-aware repulsion only at the identified vulnerable heads. Evaluated against 16 baselines, AEGIS improves both safety and utility. On SD 1.4, it reduces ASR to $\mathbf{0.00}/\mathbf{0.03}$ for in-domain violence/nudity VSA and achieves ASRs $\le \mathbf{0.09}$ on out-of-domain explicit and adversarial attacks. It preserves benign fidelity, avoids suppressing hard-negative concepts, and transfers to SD 2.1 and FLUX.1 after re-identifying the critical heads for each backbone.

Problem

• visual synonym attacks
• text-to-image models
• safety-utility dilemma
• unsafe semantics
• jailbreak

Innovation

• visual synonym attacks
• mechanism-guided defense
• semantic-injecting attention heads
• inference-time safety
• text-to-image alignment