Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replay

2026-01-15

AI Summary

This work addresses the vulnerability of large language models to novel adversarial “jailbreaking” attacks and the limited generalization of existing safety alignment methods based on static red-teaming data. To overcome this, the paper proposes Safe Self-Play (SSP), a framework that enables a single model to autonomously assume both attacker and defender roles within a reinforcement learning loop, dynamically generating and defending against jailbreak attempts. SSP incorporates a reflective experience replay mechanism and leverages an Upper Confidence Bound (UCB) sampling strategy to prioritize high-difficulty failure cases, thereby enabling continuously evolving proactive safety alignment. Experimental results demonstrate that SSP significantly outperforms static adversarial training baselines across multiple benchmarks, establishing a new paradigm for aligning large language models with safety objectives.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements

Abstract

Large Language Models (LLMs) have achieved remarkable capabilities but remain vulnerable to adversarial ``jailbreak''attacks designed to bypass safety guardrails. Current safety alignment methods depend heavily on static external red teaming, utilizing fixed defense prompts or pre-collected adversarial datasets. This leads to a rigid defense that overfits known patterns and fails to generalize to novel, sophisticated threats. To address this critical limitation, we propose empowering the model to be its own red teamer, capable of achieving autonomous and evolving adversarial attacks. Specifically, we introduce Safety Self- Play (SSP), a system that utilizes a single LLM to act concurrently as both the Attacker (generating jailbreaks) and the Defender (refusing harmful requests) within a unified Reinforcement Learning (RL) loop, dynamically evolving attack strategies to uncover vulnerabilities while simultaneously strengthening defense mechanisms. To ensure the Defender effectively addresses critical safety issues during the self-play, we introduce an advanced Reflective Experience Replay Mechanism, which uses an experience pool accumulated throughout the process. The mechanism employs a Upper Confidence Bound (UCB) sampling strategy to focus on failure cases with low rewards, helping the model learn from past hard mistakes while balancing exploration and exploitation. Extensive experiments demonstrate that our SSP approach autonomously evolves robust defense capabilities, significantly outperforming baselines trained on static adversarial datasets and establishing a new benchmark for proactive safety alignment.

Problem

• jailbreak attacks
• safety alignment
• adversarial robustness
• large language models
• red teaming

Innovation

• Safety Self-Play
• Red Teaming
• Reflective Experience Replay
• Reinforcement Learning
• Adversarial Jailbreak