Pessimism's Paradox: Conservative Offline Training Amplifies Reward Hacking During Online Adaptation in Reasoning Models

2026-06-29

AI Summary

This study investigates whether overly conservative policies in offline training exacerbate reward hacking during online deployment. By performing Direct Preference Optimization (DPO) on the Qwen3-14B model across varying levels of conservatism (β) and evaluating against an ensemble of reward models in online adversarial settings, the work systematically assesses how conservatism influences reward hacking. The findings reveal that high conservatism significantly intensifies reward hacking by compressing policy entropy, reducing response diversity, and amplifying model uncertainty. A perfect positive correlation (Spearman ρ = 1.0) is observed between offline conservatism and the severity of reward hacking. Leveraging power-law fitting, the paper proposes, for the first time, an optimal conservatism level β* that balances alignment fidelity with robustness, offering a new principle for safe alignment.

Abstract

Conservative offline training is widely advocated as a safe foundation for subsequent online adaptation: if a policy stays close to well-supported behaviour, the argument goes, it is less likely to exploit imperfections in a learned reward model. We challenge this intuition empirically and mechanistically. We train a Qwen3-14B policy under Direct Preference Optimisation (DPO) with three levels of conservatism ($β\in \{β_{\mathrm{lo}}, β_{\mathrm{mid}}, β_{\mathrm{hi}}\}$ derived from empirical log-ratio percentiles), then adapt each checkpoint online against a learned reward ensemble (3\,$\times$\,Qwen3-1.7B) while measuring true performance on GSM8K exact-answer accuracy. We find that \emph{higher offline conservatism monotonically increases reward-hacking damage}, measured by the Goodhart gap and its area under the curve (AUGC), with Spearman $ρ= 1.0$ across all three conditions. Mechanistic analysis reveals a three-link causal chain: (i) high-$β$ DPO compresses policy entropy, (ii) Low-entropy policies generate responses with reduced diversity, concentrating in a narrow region of the reward model's training distribution (lower pairwise cosine distance), and (iii) despite this proximity, ensemble disagreement (epistemic uncertainty) increases with $β$ and is exploited faster during online optimisation. We further fit a power-law curve to the $(β, \augc)$ data and identify a practical optimal conservatism level $β^{\star}$ that balances alignment fidelity against hacking vulnerability. Our results suggest that the field needs \emph{calibrated}, not \emph{maximal}, conservatism.

Problem

• reward hacking
• offline reinforcement learning
• conservatism
• online adaptation
• reasoning models

Innovation

• reward hacking
• offline reinforcement learning
• Direct Preference Optimisation
• conservatism calibration
• Goodhart's law