Inoculation Midtraining with Learned Neologisms

2026-09-14

AI Summary

研究通过在中训阶段引入隔离标记来教授模型识别不安全行为，以减少后续训练中的不良属性泛化，同时保留良性数据特性。

Abstract

Large language models (LLMs) often learn both desirable and undesirable properties during post-training. We study whether midtraining, an earlier training stage, can shape which of these properties later generalise. We introduce Inoculation Midtraining, a technique that teaches a base model that unsafe behaviour belongs to a designated <quarantine_token> context, as indicated by the <quarantine_token> neologism (a new token) introduced during midtraining, and then post-trains the model on unsafe data within that context. We then evaluate the model outside the context, with the <quarantine_token> neologism excluded from the system prompt. Across supervised fine-tuning and reinforcement learning post-training regimes, we find that Inoculation Midtraining can reduce misalignment while preserving the transfer of benign data properties (e.g., speaking in German or Shakespearean prose). However, our approach does not outperform standard Inoculation Prompting, is sensitive to training configuration, and produces a leaky boundary that nearby contextual cues can reactivate. These results show that inoculation with a learned association introduced via midtraining can shape selective generalisation. Still, more work is needed before this approach can become a load-bearing component in a developer's safety framework.

Problem

• large language models
• midtraining
• generalisation
• unsafe behaviour
• inoculation

Innovation

• Inoculation Midtraining
• neologism
• <quarantine_token>
• misalignment