OTTER: A Red-Teaming System for Toxicity-Evading Jailbreak Prompt Optimization

2026-06-18

AI Summary

Current safety mechanisms in large language models rely on surface-level detection of toxic words, erroneously assuming that malicious intent necessarily manifests through explicit harmful phrasing—an assumption easily circumvented. This work proposes OTTER, a black-box red-teaming framework that decouples surface toxicity from genuine malicious intent by substituting as few as five tokens on average, thereby generating effective jailbreaking prompts within standard API constraints. OTTER integrates token-level evolutionary rewriting with adversarial prompt generation, requiring no internal model access. Experiments across 457 AdvBench prompts and four GPT-family models demonstrate that the attack success rate increases from 7.0% to 84.0%. This study provides the first quantitative evidence linking toxicity evasion to attack efficacy and introduces a practical classifier hardening strategy to mitigate such vulnerabilities.

Abstract

Production LLMs increasingly rely on toxicity-based moderation filters as a primary defense, assuming that harmful intent correlates with toxic surface wording. We show this assumption is fundamentally brittle: surface toxicity and adversarial intent can be decoupled by replacing as few as five tokens. We present OTTER (Obfuscated Toxicity-Evading Token Evolution for Rewriting), a black-box red-teaming framework requiring only standard API access, directly targeting the practical constraints of industry security audits. Evaluated on 457 AdvBench prompts across four GPT models, OTTER raises average ASR from 7.0% to 84.0%. We further provide the first quantitative analysis of the toxicity--bypass relationship and a per-category breakdown, translating our findings into actionable recommendations for classifier hardening in production deployments.

Problem

• toxicity evasion
• jailbreak prompts
• red-teaming
• moderation filters
• adversarial intent

Innovation

• red-teaming
• toxicity evasion
• jailbreak prompt optimization
• black-box attack
• LLM moderation