On-Policy Consistency Training Improves LLM Safety with Minimal Capability Degradation

2026-05-20

AI Summary

Current aligned large language models still suffer from sycophancy, vulnerability to jailbreak attacks, and insufficient safety warnings, while conventional consistency training exhibits poor generalization and often leads to capability degradation. This work proposes Online Policy Consistency Training (OPCT), which for the first time shifts consistency training from offline supervised fine-tuning to an online policy optimization framework grounded in the model’s own responses. OPCT dynamically constructs supervision signals through contrastive prompt generation and a self-supervised consistency objective. Experimental results demonstrate that OPCT reduces sycophancy to 8.1% (compared to 11.2% with standard supervised fine-tuning), improves jailbreak defense success rate to 99% (versus 87% with supervised fine-tuning), and effectively prevents performance degradation on benchmarks such as MATH-500, thereby substantially enhancing both safety and generalization.

Abstract

Aligned models can misbehave in several ways: they are often sycophantic, fall victim to jailbreaks, or fail to include appropriate safety warnings. Consistency training is a promising new alignment paradigm to mitigate such failures by training invariants into the model using contrastive input pairs. Existing consistency training procedures generate the supervision signal once, offline, and use supervised fine-tuning (SFT) to update the model. Unfortunately, the resulting models tend to merely memorize the surface forms of the training distribution and thus generalize poorly and regress in their capabilities. We introduce On-Policy Consistency Training (OPCT), a new consistency training approach where the objective is computed over the model's own responses to prompts, supervised by itself conditioned on corresponding contrastive prompts. We evaluate OPCT on three safety axes: sycophancy, jailbreaking, and safety awareness. Across three model families, OPCT outperforms its SFT counterpart on all safety desiderata. It nearly halves the sycophancy rate relative to baseline (8.1% vs. 15.4%, compared to 11.2% for SFT). Under an adaptive per-target attacker, OPCT holds jailbreak defense success near 99% on held-out jailbreak behaviors, whereas SFT achieves 87% on average. On safety awareness, OPCT outperforms SFT in two out of three models, and matches it on the other. OPCT also largely avoids the capability regressions that SFT induces, such as a 28-point drop on MATH-500. Our results suggest that consistency training is best implemented as OPCT rather than as SFT, especially when generalization beyond the training distribution is desired.

Problem

• LLM safety
• sycophancy
• jailbreaking
• safety awareness
• capability degradation

Innovation

• On-Policy Consistency Training
• LLM Safety
• Contrastive Prompts
• Supervised Fine-Tuning
• Generalization