Towards Effective Prompt Stealing Attack against Text-to-Image Diffusion Models

2025-08-09

AI Summary

Prompt engineering in text-to-image diffusion models constitutes critical intellectual property, yet remains vulnerable to reverse-engineering attacks; existing methods suffer from poor generalizability and weak adaptability. This paper proposes Prometheus, a training-free, search-based prompt inversion framework. Leveraging a lightweight surrogate model, Prometheus integrates dynamic modifier generation, context-aware matching ranking, and greedy feedback-driven search to achieve efficient cross-model (e.g., DALL·E, Midjourney, Leonardo.ai) and multi-scenario (including platform-deployed images) prompt reconstruction. Crucially, it requires no access to target-model gradients or fine-tuning, ensuring strong robustness and low computational overhead. Evaluated on real-world platforms—including PromptBase and AIFrog—Prometheus improves average attack success rate by 25.0% over baselines and maintains high effectiveness against prevalent defense mechanisms.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Text-to-Image (T2I) models, represented by DALL$cdot$E and Midjourney, have gained huge popularity for creating realistic images. The quality of these images relies on the carefully engineered prompts, which have become valuable intellectual property. While skilled prompters showcase their AI-generated art on markets to attract buyers, this business incidentally exposes them to extit{prompt stealing attacks}. Existing state-of-the-art attack techniques reconstruct the prompts from a fixed set of modifiers (i.e., style descriptions) with model-specific training, which exhibit restricted adaptability and effectiveness to diverse showcases (i.e., target images) and diffusion models. To alleviate these limitations, we propose Prometheus, a training-free, proxy-in-the-loop, search-based prompt-stealing attack, which reverse-engineers the valuable prompts of the showcases by interacting with a local proxy model. It consists of three innovative designs. First, we introduce dynamic modifiers, as a supplement to static modifiers used in prior works. These dynamic modifiers provide more details specific to the showcases, and we exploit NLP analysis to generate them on the fly. Second, we design a contextual matching algorithm to sort both dynamic and static modifiers. This offline process helps reduce the search space of the subsequent step. Third, we interact with a local proxy model to invert the prompts with a greedy search algorithm. Based on the feedback guidance, we refine the prompt to achieve higher fidelity. The evaluation results show that Prometheus successfully extracts prompts from popular platforms like PromptBase and AIFrog against diverse victim models, including Midjourney, Leonardo.ai, and DALL$cdot$E, with an ASR improvement of 25.0%. We also validate that Prometheus is resistant to extensive potential defenses, further highlighting its severity in practice.

Problem

• Stealing valuable prompts from text-to-image models
• Overcoming limitations of existing prompt attack techniques
• Enhancing adaptability to diverse showcases and models

Innovation

• Training-free, proxy-in-the-loop, search-based attack
• Dynamic modifiers generated via NLP analysis
• Contextual matching and greedy search algorithm