Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models

2026-07-02

AI Summary

This work addresses the trade-off between safety and helpfulness in large language model alignment, where excessive refusal often undermines utility, and existing constructive safety approaches suffer from poor out-of-distribution generalization or overgeneralized safety reasoning. To overcome these limitations, we propose a reinforcement learning–based constructive safety alignment framework that innovatively integrates a Zero-RL paradigm with a multi-stage reinforcement learning strategy. This approach avoids indiscriminate refusals while generating responses that are both safe and helpful. Empirical evaluations demonstrate that our method substantially improves the joint performance of safety and usefulness, consistently outperforming Qwen3-14B and Oyster-I across multiple benchmarks, and achieving safety levels comparable to those of much larger models such as Qwen3-Max and Qwen3.5-397B.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities across diverse applications, yet ensuring their simultaneous safety, helpfulness, and trustworthiness remains a persistent challenge. Conventional refusal-oriented alignment strategies mitigate harmful content generation but systematically fail to serve legitimate user needs, often withholding information that could safely and constructively address the underlying intent of sensitive queries. Building upon the constructive safety paradigm pioneered by Oyster-I, which moves beyond blanket refusal toward thoughtful, response-oriented safety alignment, we identify two critical limitations of its Supervised Fine-Tuning (SFT)-based scheme: insufficient safety generalization to out-of-distribution scenarios and a phenomenon we term safety chain-of-thought (CoT) over-generalization, wherein safety-oriented reasoning patterns are excessively applied to benign queries, degrading helpfulness and user experience. To address these limitations, we propose Oyster-II, a reinforcement learning (RL)-based constructive safety alignment framework that adopts a Zero-RL paradigm combined with a multi-stage reinforcement learning strategy.Evaluated across extensive benchmarks, Oyster-II comprehensively surpasses both Qwen3-14B and its predecessor Oyster-I on safety dimensions, achieving cross-scale performance comparable to Qwen3-Max and Qwen3.5-397B.

Problem

• safety alignment
• constructive safety
• large language models
• helpfulness
• safety generalization

Innovation

• Reinforcement Learning
• Constructive Safety
• Safety Alignment
• Chain-of-Thought Over-Generalization
• Zero-RL