PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models

2026-06-24

AI Summary

This work proposes PolicyAlign, a novel framework that enables safety alignment of large language models directly from natural language safety policies without relying on costly human annotations. Addressing the inflexibility of existing alignment methods in adapting to dynamically evolving safety guidelines, PolicyAlign introduces policy-guided instruction synthesis to generate violation examples, combined with a policy-aware self-distillation mechanism and a policy-sensitive filtering strategy to efficiently refine model behavior. Experimental results demonstrate that PolicyAlign significantly enhances model safety across multiple architectures while maintaining low over-rejection rates and preserving general capabilities. Moreover, the approach successfully generalizes to high-stakes domains such as healthcare, legal, and financial applications, showcasing its robustness and practical applicability.

Abstract

Safety alignment of large language models (LLMs) typically depends on high-quality supervision data, such as safe demonstrations or preference pairs. However, in real-world deployment, emerging safety requirements are often specified as natural-language policies, while corresponding supervision data may be costly, delayed, or unavailable. This creates a mismatch between rapidly evolving safety policies and conventional data-driven alignment methods. To address this, we propose PolicyAlign, a simple yet effective framework for directly aligning LLMs with safety policies. Given a safety policy, PolicyAlign first synthesizes policy-violating instructions and then performs on-policy self-distillation to internalize policy-guided behavior. To improve training stability and data efficiency, we further introduce Policy-Sensitive Filtering, which selects instructions where the policy induces the largest behavioral shift. Experiments across multiple models show that PolicyAlign consistently improves safety while maintaining low over-refusal and preserving general capabilities. PolicyAlign also generalizes to medical, legal, and financial safety scenarios, highlighting its potential as a scalable and maintainable approach to policy-based LLM safety alignment. The code is released at https://github.com/Qwen-Applications/PolicyAlign.

Problem

• safety alignment
• large language models
• natural-language policies
• supervision data
• policy mismatch

Innovation

• Policy-Based Alignment
• On-Policy Self-Distillation
• Policy-Sensitive Filtering
• Safety Alignment
• Large Language Models