JailbreakSkill: Scaling Automated Red-Teaming with Reusable and Ever-Evolving Skills

2026-08-17

AI Summary

This study addresses the fragmentation, poor reusability, and scalability bottlenecks in automated red-teaming by proposing a skill-centric evolutionary framework. Through modular encapsulation, agent adaptation, and feedback-driven diagnostic optimization, we construct a reusable, continuously evolving attack skill library that enables closed-loop adversarial learning and automatic discovery of novel strategies. Experiments demonstrate that this approach increases Attack Success Rates (ASR) on AdvBench and HarmBench by 17.5% and 13.4%, respectively, with a substantial 48.6% improvement against GPT-5.4. Furthermore, the method successfully generates generalizable novel attack strategies, significantly enhancing both the systematicity and efficiency of red-teaming evaluations.

Abstract

Automated red-teaming has produced a growing collection of attack strategies, yet they typically remain scattered across prompts and workflows, making them difficult to systematically integrate, reuse, and improve at scale. We introduce \textsc{JailbreakSkill}, a skill-centric framework for scaling automated red-teaming through reusable and continuously evolving attack capabilities. \textsc{JailbreakSkill} packages existing attack strategies into modular, agent-ready skills that can be directly reused and adaptively selected across tasks and target models. Beyond reuse, it closes the loop between attacking and learning: attack experience is used to diagnose, refine, combine, and discover new skills, which are added back to an ever-growing skill library. This evolution lifts macro-average ASR by 17.5 percentage points on AdvBench and 13.4 points on HarmBench, including a 48.6-point gain against GPT-5.4 on AdvBench, while yielding novel attack strategies such as reframing a direct request as an unfinished document-completion task. Several evolved skills also generalize to unseen prompts and target models without further adaptation. Our code is available at https://github.com/BattleWen/JailbreakSkill.

Problem

• Automated Red-Teaming
• Attack Strategy Reuse
• Scalability
• Jailbreak

Innovation

• Automated Red-Teaming
• Reusable Skills
• Skill Evolution
• Modular Framework
• Attack-Learning Loop