SHARD: Safe and Helpful Alignment via Self-Reframing Distillation

2026-06-13

AI Summary

Large language models often over-defend against sensitive prompts, either refusing to respond or generating generic, safe-but-unhelpful replies, thereby struggling to balance safety and usefulness. This work proposes a self-reconstructive distillation approach that rewrites sensitive prompts according to philosophical guiding principles to explicitly convey benign intent, then leverages the model’s own reconstructed data to refine responses toward both safety and helpfulness. Through instruction tuning, this method achieves intrinsic alignment between safety and utility without relying on larger teacher models—the first such endogenous alignment strategy. Experiments demonstrate that, on the DNA and LINGUASAFE English subsets, the approach significantly enhances the helpfulness of mainstream model families while maintaining high safety standards, matching the performance of distillation-based methods that employ larger teacher models.

Abstract

Large language models often struggle with sensitive prompts. They may refuse outright, provide generic safety boilerplate, or fail to address the user's legitimate informational needs that can be answered safely. We introduce SHARD, a self-reframing distillation method to improve safe-helpfulness. It first rewrites sensitive prompts to surface benign intent using philosophical guidelines, then reframes its original responses into safe, more helpful ones, and finally fine-tunes the model on its self-reframed responses. Across DNA and the English subset of LINGUASAFE, SHARD improves helpfulness for most model families while preserving safety. It also remains competitive with distillation from a larger teacher model, suggesting that models can internalize safe and helpful behavior elicited from their own. Warning: This paper contains content that may be offensive or harmful.

Problem

• sensitive prompts
• safe-helpfulness
• large language models
• alignment
• harmful content

Innovation

• self-reframing distillation
• safe-helpfulness
• prompt rewriting
• alignment
• large language models