Discretizing Reward Models

2026-06-19

AI Summary

This work addresses the excessive sensitivity of existing neural reward models to semantically equivalent responses, a flaw that often triggers reward gaming in reinforcement learning and degrades policy performance. To mitigate this issue, the authors propose a training-free discretization method that leverages Monte Carlo Dropout to generate reward clusters, mapping continuous rewards to discrete values while preserving discriminative capacity. To better evaluate reward models, they introduce two novel metrics: discriminability and specificity. Empirical results demonstrate that the proposed approach significantly suppresses reward gaming and enhances policy quality across both control and natural language reinforcement learning environments.

Abstract

Despite their widespread use, the role of reward models in shaping reinforcement learning is poorly understood. Reward models offer a tempting promise: they automatically estimate response quality in the absence of verifiers or human judges. Unlike "verifiable rewards" which typically produce binary scores, reward models typically produce continuous scores, allowing them to be sensitive to fine-grained differences in responses. However, we show this apparent strength is a serious weakness: many popular reward models are oversensitive, assigning different scores to equally good responses. Theoretically, we show that seemingly perfect reward models can be highly oversensitive; empirically, this oversensitivity can lead to bad policies. In place of existing notions of "reward model accuracy," we propose evaluating reward models using distinct measures of "discriminative ability" and "specificity" (the complement of oversensitivity). As a solution, we describe a training-free algorithm that uses Monte Carlo dropout on any neural reward model to produce discrete reward clusters. Theoretically, we prove there exist discretizations that reduce oversensitivity at minimal expense of discriminative ability; empirically we show, in both controlled and natural RL settings, that discretizing rewards leads to less reward hacking and better policies than training on the original rewards.

Problem

• reward models
• oversensitivity
• reinforcement learning
• discretization
• reward hacking

Innovation

• reward model discretization
• oversensitivity
• Monte Carlo dropout
• discriminative ability
• specificity