Rapid Poison: Practical Poisoning Attacks Against the Rapid Response Framework

2026-06-15

AI Summary

This work addresses the vulnerability of the Rapid Response framework in continual learning to data poisoning attacks. Under a strict threat model that permits adversaries to manipulate only jailbreak samples, the authors propose a novel prompt-injection-based poisoning method. By synthesizing poisoned examples, the approach enables two attack modes: first, causing the model to misclassify benign inputs as jailbreak samples containing specific features; second, implanting a conceptual backdoor that allows trigger-embedded jailbreak samples to evade detection. The key innovation lies in an “omission attack” mechanism, which exploits the model’s erroneous associations with missing concepts to achieve highly effective label flipping. Experimental results demonstrate that with merely a 1% poisoning rate, the method achieves a 100% false positive rate and a 96% false negative rate, substantially degrading the framework’s jailbreak detection capability.

Abstract

The Rapid Response (RR) framework, deployed in production systems, including Anthropic's ASL-3 safeguards, continuously improves jailbreak-detection classifiers. When new jailbreaks emerge that bypass these classifiers, Rapid Response generates synthetic variants for training, helping the model generalize from the new attacks and quickly adapt. We reveal that prompt injection can infiltrate this pipeline to deliver poisoned samples into the classifier's training set, enabling two attack objectives: (I) targeted poisoning attacks that create false positives on harmless samples by categorizing them as a jailbreak, with a specific desired feature (e.g., certain formatting, subject, or keyword), (II) concept-based backdoor attacks that induce false negatives on jailbreak inputs, generalizing even to jailbreaks from attack strategies the defender explicitly trained against, when the backdoor trigger is present. Importantly, our threat model restricts adversaries to modifying only jailbreak samples (not benign data or labels), a constraint unexplored by prior work that makes the second objective particularly challenging. We address this with Omission Attack, which exploits a new phenomenon: when training on concept-absent unsafe samples, the classifier misassociates that concept's presence with the safe label. Both attacks cause substantial and in some cases near-complete label flipping at only a 1% poisoning rate, achieving up to 100% false positive rates and up to 96% false negative rates.

Problem

• poisoning attacks
• jailbreak detection
• backdoor attacks
• Rapid Response framework
• false positives

Innovation

• poisoning attack
• Rapid Response framework
• backdoor attack
• Omission Attack
• jailbreak detection