Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking

2026-09-14

AI Summary

研究探讨了合成文档微调（SDF）是否能防止模型在后续训练中出现奖励黑客导致的意外误对齐，结果表明SDF虽然可以改变模型行为，但不能有效阻止误对齐。

Abstract

Recent work shows that models that learn to reward hack on RL environments can become broadly misaligned, and that reframing reward hacking as acceptable behavior during training (inoculation prompting, or IP) blocks this generalization. We ask whether synthetic document finetuning (SDF) can inoculate a model against future training we don't intervene on. We add synthetic documents framing reward hacking as acceptable behavior to a model's midtraining corpus, and then train these models with RL on exploitable environments, teaching them to reward hack. Behaviorally, midtraining succeeds: models describe reward hacking favorably and are more approving of reward-hacking outputs they produce. However, they show strong EM after learning to reward hack, while IP in the same setting prevents EM. We show that SDF can predictably steer downstream generalization when inserting new associations, but struggles and has unpredictable effects when overriding existing associations, such as that between reward hacking and misalignment that produces EM. Our results suggest that, at the scales we test, SDF can make a model appear aligned with desired beliefs while steering its generalization from later training in unintended ways.

Problem

• Synthetic document finetuning
• Reward hacking
• Emergent misalignment

Innovation

• Synthetic Document Finetuning
• Reward Hacking
• Emergent Misalignment
• Inoculation Prompting