PixJail: Self-Evolving Paper-to-Pipeline Reproduction for Text-to-Image Jailbreak Evaluation

2026-06-22

AI Summary

This work addresses the reproducibility and fairness challenges in evaluating text-to-image (T2I) jailbreak methods, which often rely on complex, multi-stage pipelines. To overcome this, the authors propose a self-evolving agent framework that enables code-free reproduction of T2I jailbreak approaches for the first time. The framework employs a unified contract interface to automatically construct attack modules and evaluation pipelines, and incorporates a self-evolving memory mechanism that integrates paper abstracts, attack templates, failure cases, and versioned artifacts to facilitate knowledge reuse. Using this framework, the authors successfully reproduce 11 representative jailbreak methods, achieving an average error of only 2.1% (median error: 0%) under original settings, thereby substantially reducing manual reproduction effort and enhancing both reproducibility and fairness in T2I jailbreak evaluation.

Abstract

As Text-to-Image (T2I) jailbreak techniques evolve rapidly, existing benchmarks and reproduction workflows often struggle to keep pace. More importantly, T2I jailbreak evaluation is not a single prompt-level test, but a pipeline-level problem shaped by multiple stages, including prompt transformation, image generation, safety filtering, and multimodal judging. This makes results across papers difficult to reliably reproduce and fairly compare. To bridge this gap, we propose PixJail, a self-evolving paper-to-pipeline agent framework for reproducible T2I jailbreak evaluation. Given a T2I jailbreak paper and optional reference code, PixJail rapidly constructs a paper-specific attack module and a runnable evaluation pipeline under a unified contract, while faithfully reproducing the original experimental results. PixJail further maintains a memory bank that stores paper digests, attack evolution patterns, reusable templates, failure cases, and versioned artifacts, enabling future reproduction efforts to reuse prior experience. We reproduce eleven representative T2I jailbreak methods, including both code-available and code-unavailable papers. Under their original settings, our framework accurately recovers prior results with minimal error (2.1\% average, 0\% median). We hope that PixJail can serve as a unified foundation for future T2I jailbreak reproduction and evaluation, significantly reducing manual effort.

Problem

• Text-to-Image Jailbreak
• Reproducibility
• Evaluation Pipeline
• Benchmarking
• Multimodal Safety

Innovation

• self-evolving
• paper-to-pipeline
• reproducible evaluation
• text-to-image jailbreak
• memory bank