In-Context Experience Replay Facilitates Safety Red-Teaming of Text-to-Image Diffusion Models

2024-11-25

arXiv.org

AI Summary

Systematic, model-agnostic red-teaming tools for security evaluation of text-to-image (T2I) models remain absent. Method: We propose the first zero-shot black-box red-teaming framework grounded in contextual experience replay, integrating large language model (LLM) prompt engineering, multi-armed bandit optimization, and semantic similarity constraints to automatically generate semantically coherent yet harmful test prompts. Its core innovation lies in leveraging historically successful jailbreaking instances to guide discovery of novel vulnerabilities, thereby uncovering transferable patterns in prompt-based adversarial attacks. Contribution/Results: Evaluated across multiple state-of-the-art T2I models, our method achieves up to a 3.2× improvement in vulnerability detection rate while maintaining >92% semantic fidelity—significantly outperforming existing prompt-based attack approaches.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Text-to-image (T2I) models have shown remarkable progress, but their potential to generate harmful content remains a critical concern in the ML community. While various safety mechanisms have been developed, the field lacks systematic tools for evaluating their effectiveness against real-world misuse scenarios. In this work, we propose ICER, a novel red-teaming framework that leverages Large Language Models (LLMs) and a bandit optimization-based algorithm to generate interpretable and semantic meaningful problematic prompts by learning from past successful red-teaming attempts. Our ICER efficiently probes safety mechanisms across different T2I models without requiring internal access or additional training, making it broadly applicable to deployed systems. Through extensive experiments, we demonstrate that ICER significantly outperforms existing prompt attack methods in identifying model vulnerabilities while maintaining high semantic similarity with intended content. By uncovering that successful jailbreaking instances can systematically facilitate the discovery of new vulnerabilities, our work provides crucial insights for developing more robust safety mechanisms in T2I systems.

Problem

• Evaluates safety mechanisms in text-to-image models
• Generates problematic prompts using LLMs and bandit optimization
• Identifies vulnerabilities without internal access or retraining

Innovation

• Utilizes Large Language Models
• Employs bandit optimization algorithm
• Generates interpretable problematic prompts