STACK: Adversarial Attacks on LLM Safeguard Pipelines

2025-06-30

AI Summary

State-of-the-art AI systems rely on multi-layered safety pipelines—such as few-shot prompt classifiers—to mitigate catastrophic misuse, yet these defenses lack systematic security evaluation and adversarial analysis. Method: We conduct the first red-teaming study targeting LLM safety pipelines and propose STaged AttaCK (STACK), a black-box attack framework designed to bypass prompt-based classifiers while enabling cross-model transfer. Contribution/Results: On the ClearHarm benchmark, STACK achieves a 71% success rate in direct attacks and 33% in transfer attacks; critically, it fully neutralizes baseline defenses, reducing their effectiveness to 0%. Our work exposes critical vulnerabilities in current multi-layered safety architectures and establishes the first empirical benchmark and methodological advance for robust AI safety engineering.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements

Abstract

Frontier AI developers are relying on layers of safeguards to protect against catastrophic misuse of AI systems. Anthropic guards their latest Claude 4 Opus model using one such defense pipeline, and other frontier developers including Google DeepMind and OpenAI pledge to soon deploy similar defenses. However, the security of such pipelines is unclear, with limited prior work evaluating or attacking these pipelines. We address this gap by developing and red-teaming an open-source defense pipeline. First, we find that a novel few-shot-prompted input and output classifier outperforms state-of-the-art open-weight safeguard model ShieldGemma across three attacks and two datasets, reducing the attack success rate (ASR) to 0% on the catastrophic misuse dataset ClearHarm. Second, we introduce a STaged AttaCK (STACK) procedure that achieves 71% ASR on ClearHarm in a black-box attack against the few-shot-prompted classifier pipeline. Finally, we also evaluate STACK in a transfer setting, achieving 33% ASR, providing initial evidence that it is feasible to design attacks with no access to the target pipeline. We conclude by suggesting specific mitigations that developers could use to thwart staged attacks.

Problem

• Evaluating security of AI safeguard pipelines against attacks
• Developing novel classifier to reduce attack success rates
• Introducing staged attack method to bypass existing defenses

Innovation

• Novel few-shot-prompted classifier reduces attack success
• STACK procedure achieves 71% black-box attack success
• Transfer attack shows 33% success without access