Semantic-level Backdoor Attack against Text-to-Image Diffusion Models

2026-02-03

AI Summary

Existing backdoor attacks on text-to-image diffusion models typically rely on fixed textual triggers and single-target objectives, making them vulnerable to input-level defenses. This work proposes a novel approach that implants backdoors at the semantic representation level by distilling edits into the key-value projection matrices of cross-attention layers. Integrating a continuous semantic-region triggering mechanism, multi-entity backdoor targets, and semantic regularization constraints, the method enables diverse prompts sharing the same semantic structure to activate the backdoor. The approach achieves 100% attack success rate while preserving high image generation quality and demonstrates strong robustness against state-of-the-art input-level defenses, significantly enhancing both the stealthiness and generalization capability of the implanted backdoor.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Text-to-image (T2I) diffusion models are widely adopted for their strong generative capabilities, yet remain vulnerable to backdoor attacks. Existing attacks typically rely on fixed textual triggers and single-entity backdoor targets, making them highly susceptible to enumeration-based input defenses and attention-consistency detection. In this work, we propose Semantic-level Backdoor Attack (SemBD), which implants backdoors at the representation level by defining triggers as continuous semantic regions rather than discrete textual patterns. Concretely, SemBD injects semantic backdoors by distillation-based editing of the key and value projection matrices in cross-attention layers, enabling diverse prompts with identical semantic compositions to reliably activate the backdoor attack. To further enhance stealthiness, SemBD incorporates a semantic regularization to prevent unintended activation under incomplete semantics, as well as multi-entity backdoor targets that avoid highly consistent cross-attention patterns. Extensive experiments demonstrate that SemBD achieves a 100% attack success rate while maintaining strong robustness against state-of-the-art input-level defenses.

Problem

• backdoor attack
• text-to-image diffusion models
• semantic triggers
• input defenses
• cross-attention

Innovation

• semantic-level backdoor
• diffusion models
• cross-attention editing
• distillation-based injection
• multi-entity targets