Regime-Conditional Verification: Correctness Estimation for Adapting and Monitoring Safety Classifiers

2026-08-14

AI Summary

This study addresses performance degradation in safety classifiers for large language models caused by policy misalignment and traffic drift. We propose RCV, a lightweight wrapper framework that leverages internal representations to estimate prediction correctness and perform selective corrections, while providing unlabeled drift detection signals to trigger a retraining-free maintenance loop. Experiments demonstrate that RCV significantly improves policy adherence across all tested configurations, recovering 81% of missed unsafe content. The framework effectively defends against adversarial attacks and mitigates distribution shifts, achieving adaptive safety enhancement that aligns with target policies and monitors distributional changes without requiring model retraining.

Abstract

Safety classifiers deployed with large language models often fail for two reasons: their decisions reflect the policy learned during training rather than the deployer's desired policy, and their performance degrades as deployment traffic evolves. We present Regime-Conditional Verification (RCV), a lightweight wrapper that adapts an off-the-shelf safety classifier without retraining it. RCV estimates, from the classifier's internal representations, the probability that each prediction disagrees with the deployer's policy, and selectively corrects predictions likely to be wrong. The same correctness estimates also provide a label-free signal for detecting distribution shift, enabling a maintenance loop that updates the correctness estimation layer and resorts to classifier fine-tuning only when necessary. Across three off-the-shelf safety classifiers and two benchmark datasets, RCV improves adherence to the deployer's policy in every classifier-dataset combination, catching up to 0.81 of previously missed unsafe content without modifying the underlying classifier. In a deployment study with ten attack campaigns, each a harm category held out of RCV's training, RCV detects every campaign in a dedicated injection panel; in the maintenance census most drift episodes are repaired without updating the classifier, and the fine-tune is reserved for the residual episodes that repair does not restore.

Problem

• Safety Classifiers
• Policy Alignment
• Distribution Shift
• Correctness Estimation
• Large Language Models

Innovation

• Regime-Conditional Verification
• Safety Classifiers
• Correctness Estimation
• Distribution Shift Detection
• Label-free Monitoring