THINKSAFE: Self-Generated Safety Alignment for Reasoning Models

2026-01-30

AI Summary

This work addresses the vulnerability of large reasoning models in reinforcement learning alignment, where excessive emphasis on instruction following compromises robustness against harmful prompts, and existing external-teacher distillation methods often distort the model’s original reasoning distribution. To overcome these limitations, we propose a self-generated safety alignment framework that requires no external teacher. By introducing a lightweight rejection-guided mechanism, our approach activates the model’s intrinsic safety knowledge to autonomously generate safe reasoning trajectories that conform to its native distribution, followed by distribution-preserving fine-tuning. This method is the first to leverage a model’s latent safety capabilities for alignment, achieving significant safety improvements on DeepSeek-R1-Distill and Qwen3 while maintaining reasoning performance comparable to GRPO and substantially reducing computational costs.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Large reasoning models (LRMs) achieve remarkable performance by leveraging reinforcement learning (RL) on reasoning tasks to generate long chain-of-thought (CoT) reasoning. However, this over-optimization often prioritizes compliance, making models vulnerable to harmful prompts. To mitigate this safety degradation, recent approaches rely on external teacher distillation, yet this introduces a distributional discrepancy that degrades native reasoning. We propose ThinkSafe, a self-generated alignment framework that restores safety alignment without external teachers. Our key insight is that while compliance suppresses safety mechanisms, models often retain latent knowledge to identify harm. ThinkSafe unlocks this via lightweight refusal steering, guiding the model to generate in-distribution safety reasoning traces. Fine-tuning on these self-generated responses effectively realigns the model while minimizing distribution shift. Experiments on DeepSeek-R1-Distill and Qwen3 show ThinkSafe significantly improves safety while preserving reasoning proficiency. Notably, it achieves superior safety and comparable reasoning to GRPO, with significantly reduced computational cost. Code, models, and datasets are available at https://github.com/seanie12/ThinkSafe.git.

Problem

• safety alignment
• reasoning models
• harmful prompts
• compliance
• distributional discrepancy

Innovation

• self-generated alignment
• safety steering
• reasoning models
• refusal guidance
• distribution shift