Automated jailbreak attack targeting multiple defense strategies

2026-06-15

AI Summary

Despite the deployment of multi-layered defense mechanisms, large language models remain vulnerable to adversarial prompt attacks, and existing evaluation methods suffer from low efficiency and poor generalization. This work proposes UNIATTACK, a novel defense-oriented attack construction framework that enables one-shot, cross-model, and cross-defense black-box jailbreaking without iterative tuning. By integrating high-impact adversarial feature extraction, a dedicated attacker large language model, and automated template synthesis, UNIATTACK significantly enhances both the generality and efficiency of adversarial attacks. Experimental results demonstrate that the proposed method improves average attack success rates by 64.63%–248.82% over baseline approaches across multiple defended models, while reducing computational costs to merely 0.03%–4.96% of those required by baselines.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks. However, their safety remains a critical concern due to their susceptibility to adversarial prompt-based attacks. In this paper, we present UNIATTACK, an adversarial testing framework designed from a defense-oriented perspective to systematically construct effective black-box attack prompts. Unlike prior approaches that rely on static templates or iterative model-specific tuning, UNIATTACK extracts minimal but high-impact attack features from diverse existing attacks, optimizes them via a specialized attacker LLM, and composes them into flexible templates through automated refinement process. This feature-centric construction enables one-shot attacks that generalize across multiple models and safety categories, providing a practical tool for assessing LLM robustness. Our evaluation results shows that compared to the baselines, UNIATTACK achieves an average attack success rate (ASR) improvement of 64.63\%-248.82\% on models deployed with multi-layered defense mechanisms and it only takes 0.03\%-4.96\% cost of the baselines. UNIATTACK artifact is available at https://anonymous.4open.science/r/UniAttack-Artifact-30F1.

Problem

• jailbreak attack
• large language models
• adversarial prompts
• defense strategies
• model robustness

Innovation

• adversarial prompt
• black-box attack
• feature-centric construction
• automated jailbreak
• LLM robustness