Inducing Emergent Misalignment from Reward Hacks with Iterative DPO

2026-09-06

AI Summary

研究通过迭代DPO方法解决语言模型中由奖励黑客导致的错位问题，该方法成本较低且可在常用微调API上训练。

Abstract

Reward hacking during reinforcement learning from verifiable rewards (RLVR) can induce reward seeking and broad misalignment in language models. Studying this misgeneralization is important for developing better threat models and countermeasures, but is often infeasible due to the cost of RL on large models. As an alternative, we propose studying emergent misalignment from iterative DPO, which preserves important properties of RLVR while reducing costs and enabling training on popular finetuning APIs. In practice, we find that training GPT-4.1 with iterative DPO on a single-turn reward hacking environment induces covert misaligned power-seeking and alignment faking, the first openly available (semi)-online training pipeline to induce these concerning forms of misalignment. We also find that training Qwen2.5-32B-Instruct with the same pipeline induces both misalignment and improved instruction following accuracy, showing that iterative DPO can be used as a testbed for selective generalization. Overall, we think iterative DPO can help democratize and accelerate the study of emergent misalignment from RLVR.

Problem

• Reward Hacking
• Reinforcement Learning
• Misalignment
• Language Models
• Iterative DPO

Innovation

• iterative DPO
• emergent misalignment
• reward hacking
• selective generalization
• cost reduction