Synthetic Persona Pretraining: Alignment from Token Zero

2026-08-13

AI Summary

This work addresses the limitation of conventional language models, whose post-pretraining alignment mechanisms often result in superficial value adherence that degrades under out-of-distribution scenarios. To overcome this, the authors propose Synthetic Persona Pretraining (SPP), a novel approach that integrates value alignment at the very beginning of pretraining. SPP injects first-person reflective texts generated from a value constitution, enabling the assistant persona to internalize normative principles from token zero. This early-stage alignment is further reinforced through subsequent role-binding fine-tuning, achieving deep and robust value alignment. Experimental results demonstrate that SPP substantially enhances the model’s adherence to ethical guidelines, improves robustness against jailbreaking attempts, and maintains stable alignment in out-of-distribution moral dilemmas. Notably, introducing alignment during early pretraining yields superior performance without compromising general task capabilities.

Abstract

As language-model-based AI is increasingly deployed in autonomous settings, aligning its goals and values with those of humans becomes critical. Today, alignment, and the assistant identity itself, are typically introduced only after pretraining, once behavioral priors are already established. This can make values a thin overlay, rather than deeply rooted, and facilitate subsequent misalignment. Pursuing a different paradigm, we introduce Synthetic Persona Pretraining (SPP), which installs the desired assistant persona from token zero in pretraining. First, we annotate pretraining documents with value-aligned first-person reflections derived from a normative value constitution. Second, we pretrain via the standard cross-entropy loss on standard pretraining documents as well as their reflections, which installs the desired persona among a multitude of other personas. Finally, we post-train on user-assistant dialogue data, which binds this desired persona to the assistant identity, a process we call persona binding. By pretraining models up to 3B parameters on 500B tokens, we show that SPP improves constitution following and jailbreak robustness, and reduces the misalignment rate in out-of-distribution moral dilemmas, while preserving capabilities. Early intervention matters: compared with alignment from token zero, introducing SPP only at the end of pretraining yields weaker constitution adherence, does not shift value priorities, and leads to less aligned choices in dilemmas. This advantage depends on persona binding and, importantly, increases with pretraining budget. Overall, our results show that shaping values early is critical for alignment and establish pretraining-time persona interventions as an effective approach to do so.

Problem

• alignment
• pretraining
• persona
• values
• misalignment

Innovation

• Synthetic Persona Pretraining
• alignment from token zero
• persona binding
• value constitution
• pretraining-time alignment