Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base model

2026-04-01

AI Summary

This work addresses a critical limitation in deliberative alignment for large language models: despite alignment efforts, student models often inherit unsafe behaviors from their base models, resulting in insufficient safety guarantees. The study is the first to identify and characterize this alignment gap and proposes an attribution-based Best-of-N (BoN) sampling method that attributes unsafe responses to the base model in latent space. During inference, this approach dynamically re-ranks candidate outputs to suppress unsafe generations. Extensive experiments across seven teacher and six student models demonstrate substantial reductions in attack success rates—28.2% on DAN, 31.3% on WildJailbreak, and 35.4% on StrongREJECT—while preserving general capabilities and maintaining safety gains even after reinforcement learning-based post-training.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements

Abstract

While the wide adoption of refusal training in large language models (LLMs) has showcased improvements in model safety, recent works have highlighted shortcomings due to the shallow nature of these alignment methods. To this end, the work on Deliberative alignment proposed distilling reasoning capabilities from stronger reasoning models, thereby instilling deeper safety in LLMs. In this work, we study the impact of deliberative alignment in language models. First, we show that despite being larger in model size and stronger in safety capability, there exists an alignment gap between teacher and student language models, which affects both the safety and general utility of the student model. Furthermore, we show that models aligned through deliberative alignment can retain unsafe behaviors from the base model despite learning the reasoning patterns of larger reasoning models. Building upon this observation, we propose a BoN sampling method that attributes the unsafe behavior back to the base LLMs in the latent space, thereby down-ranking unsafe responses to gain a meaningful improvement in model safety across multiple safety benchmarks with minimal loss in utility. In particular, across 7 teacher models and 6 student models of different classes and sizes, we show an average attack success rate (ASR) reduction of 28.2% in DAN, 31.3% in WildJailbreak and 35.4 % in StrongREJECT benchmarks. We further show that these safety gains prevail post RL training, thus highlighting the uncertainty in safety reasoning and it's explicit attribution to the base model.

Problem

• Deliberative Alignment
• model safety
• alignment gap
• unsafe behavior
• base model

Innovation

• Deliberative Alignment
• BoN sampling
• attribution of unsafe behavior
• inference-time safety
• LLM alignment gap