AutoPrompt: Automated Red-Teaming of Text-to-Image Models via LLM-Driven Adversarial Prompts

2025-10-27

AI Summary

Safety filters in text-to-image (T2I) models are vulnerable to adversarial prompt attacks, yet existing red-teaming methods rely on white-box access, produce semantically incoherent prompts, and are easily blocked by detection mechanisms. Method: We propose the first large language model (LLM)-based black-box automated red-teaming framework for T2I safety evaluation. Our approach jointly optimizes perplexity scoring, prohibited-word penalties, and black-box search through alternating optimization and lightweight LLM fine-tuning to generate semantically coherent, human-readable adversarial suffixes that evade both perplexity-based detectors and keyword-based blacklists. Contribution/Results: The method requires no gradient computation or internal model access, enabling zero-shot transferable attacks across diverse open-source and commercial T2I systems—including Leonardo.AI. It significantly improves vulnerability discovery efficiency and attack stealthiness, establishing a practical, scalable paradigm for robust T2I safety assessment.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Despite rapid advancements in text-to-image (T2I) models, their safety mechanisms are vulnerable to adversarial prompts, which maliciously generate unsafe images. Current red-teaming methods for proactively assessing such vulnerabilities usually require white-box access to T2I models, and rely on inefficient per-prompt optimization, as well as inevitably generate semantically meaningless prompts easily blocked by filters. In this paper, we propose APT (AutoPrompT), a black-box framework that leverages large language models (LLMs) to automatically generate human-readable adversarial suffixes for benign prompts. We first introduce an alternating optimization-finetuning pipeline between adversarial suffix optimization and fine-tuning the LLM utilizing the optimized suffix. Furthermore, we integrates a dual-evasion strategy in optimization phase, enabling the bypass of both perplexity-based filter and blacklist word filter: (1) we constrain the LLM generating human-readable prompts through an auxiliary LLM perplexity scoring, which starkly contrasts with prior token-level gibberish, and (2) we also introduce banned-token penalties to suppress the explicit generation of banned-tokens in blacklist. Extensive experiments demonstrate the excellent red-teaming performance of our human-readable, filter-resistant adversarial prompts, as well as superior zero-shot transferability which enables instant adaptation to unseen prompts and exposes critical vulnerabilities even in commercial APIs (e.g., Leonardo.Ai.).

Problem

• Automatically generates human-readable adversarial prompts for text-to-image models
• Bypasses safety filters via dual-evasion strategy against perplexity and blacklists
• Enables black-box red-teaming with transferable attacks on commercial APIs

Innovation

• LLM-driven adversarial suffix generation for T2I models
• Alternating optimization-finetuning pipeline for suffix refinement
• Dual-evasion strategy bypassing perplexity and blacklist filters