Internalizing Safety Understanding in Large Reasoning Models via Verification

2026-05-09

AI Summary

Current large reasoning models lack an intrinsic understanding of the safety of their own outputs, rendering them vulnerable to adversarial jailbreaking attacks. This work proposes SInternal, a novel framework that internalizes safety guidelines as the model’s self-verification capability. Specifically, through supervised training, the model is guided to critically evaluate the safety of its generated content using expert reasoning trajectories, and this safety-aware initialization is then leveraged to bootstrap a reinforcement learning process. The proposed approach substantially enhances the model’s robustness against out-of-distribution jailbreak attacks. Moreover, when used as an initialization strategy for reinforcement learning, it outperforms standard supervised fine-tuning and demonstrates strong generalization in response safety across diverse scenarios.

Abstract

While explicit Chain-of-Thought (CoT) empowers large reasoning models (LRMs), it enables the generation of riskier final answers. Current alignment paradigms primarily rely on externally enforced compliance, optimizing models to detect malicious prompts rather than evaluating the safety of their own outputs. We argue that this approach remains largely behavioral: our empirical analysis reveals that ostensibly aligned models lack intrinsic safety understanding, often failing to verify their own response safety and remaining vulnerable to adversarial jailbreaks. To address this fundamental limitation, we propose Safety Internal (SInternal), a framework that internalizes safety specifications by training LRMs exclusively on safety verification tasks to critique their own generated answers using expert reasoning trajectories. We demonstrate that learning to verify induces a strong generalization for response safety, significantly enhancing robustness against out-of-domain jailbreaks. Furthermore, when combined with reinforcement learning, SInternal serves as a superior initialization compared to standard supervised fine-tuning, suggesting that internalizing safety understanding creates a more robust foundation for alignment than merely mimicking safe behaviors. Our codes are available at https://github.com/AlphaLab-USTC/SInternal

Problem

• safety alignment
• large reasoning models
• adversarial jailbreaks
• internal safety understanding
• output verification

Innovation

• Safety Internalization
• Self-Verification
• Large Reasoning Models
• Adversarial Jailbreak Robustness
• Alignment