Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection

2026-06-17

AI Summary

This work addresses the challenge that large language models can synthesize unsafe behaviors even from seemingly benign training data, a limitation that cannot be fully mitigated by data filtering or rewriting alone. To overcome this, the authors propose a “safe reflection pretraining” mechanism, which periodically interleaves brief safety-oriented reflective prompts into the pretraining corpus to internalize self-monitoring capabilities directly within the language modeling process. This approach is further reinforced through subsequent alignment training, actively steering the model toward safe behavioral tendencies rather than merely restricting it to safe data. Experiments on a 1.7B-parameter model demonstrate that the method significantly improves safety classification accuracy, substantially reduces attack success rates during both inference and fine-tuning, and outperforms conventional data filtering and rewriting strategies in the controlled MedSafetyWorld evaluation environment.

Abstract

To achieve deeper safety alignment for large language models (LLMs), recent efforts have studied how to push safety interventions earlier into the pretraining stage, primarily by filtering unsafe data or rewriting it into safer forms. We argue that pretraining-stage alignment should go beyond making the data safe: LLMs may compose seemingly benign knowledge and capabilities into unsafe behaviors. To this end, we propose Safety Reflection Pretraining, a pretraining-stage alignment method which regularly inserts short safety reflections into pretraining corpora to integrate self-monitoring directly into language modeling, establishing a foundational capability that is subsequently reinforced by compatible post-training. Our experiments with 1.7B models pretrained on FineWeb-Edu show that Safety Reflection Pretraining improves safety classification accuracy and substantially reduces the success rates of inference-stage and finetuning attacks. Complementary to our real-world experiments, we also introduce a fully controlled synthetic environment, MedSafetyWorld, with a clear definition of safety and a reasoning structure under which models can easily generalize unsafe behaviors from safe data. Ablations in MedSafetyWorld further demonstrate a clear advantage of Safety Reflection Pretraining in preventing models from acting on unsafe behaviors generalized from safe data, compared with data filtering and rewriting. Taken together, our findings suggest that pretraining alignment should not only make the training data safe, but also shape the behaviors that models are likely to acquire from safe data.

Problem

• safety alignment
• pretraining-stage
• unsafe behavior
• large language models
• safe data

Innovation

• Safety Reflection Pretraining
• pretraining-stage alignment
• self-monitoring
• MedSafetyWorld
• safe data generalization