Modification-Considering Value Learning for Reward Hacking Mitigation in RL

2026-06-27

AI Summary

Reinforcement learning agents often exhibit "reward hacking" behaviors—achieving high observed returns while failing to accomplish the intended task—due to misspecified reward functions. This work proposes MCVL, the first practical implementation of current utility optimization: during off-policy training, it leverages a frozen bootstrapped return estimator, constructed from reward models and value functions learned via DDQN or TD3, to filter experience transitions. This mechanism effectively discards reward-hacking samples while preserving informative policy updates. Empirical results demonstrate that MCVL substantially mitigates diverse reward hacking mechanisms across four grid-world environments and three modified MuJoCo tasks, consistently improving performance on the true underlying objectives.

Abstract

Reinforcement learning agents can exploit misspecified reward signals to achieve high apparent returns while failing on the intended objective, a failure mode known as reward hacking. Existing practical defenses typically constrain policy updates to stay near a known safe reference, creating a tension between suppressing hacking and permitting legitimate improvement. We propose Modification-Considering Value Learning (MCVL), which operationalizes the theoretical idea of current utility optimization for standard value-based RL. MCVL wraps an off-policy learner and treats each incoming transition as a candidate modification: it forecasts two training paths, one that includes the transition and one that does not, and scores both with a frozen bootstrapped-return estimator derived from a learned reward model and value function. The transition is admitted only if inclusion does not decrease the score. We formalize conditions under which this filtering is both safe and permissive, and instantiate MCVL with DDQN and TD3. Across four safety-relevant gridworlds and three modified MuJoCo continuous-control tasks with diverse hacking mechanisms, MCVL mitigates reward hacking while continuing to improve the intended objective. Project website: ktolnos.github.io/mcvl/.

Problem

• reward hacking
• reinforcement learning
• reward misspecification
• safe exploration
• value learning

Innovation

• reward hacking
• value learning
• off-policy RL
• utility optimization
• safety in reinforcement learning