Uncertainty-Aware Reward Modeling for Stable RLHF

2026-06-18

AI Summary

This work addresses a critical limitation in existing reinforcement learning from human feedback (RLHF) methods: reward models often lack reliable uncertainty quantification, rendering them susceptible to noisy preference labels and prone to inducing reward hacking during policy optimization. To mitigate this, the authors propose an uncertainty-aware reward modeling framework that, for the first time, integrates calibrated uncertainty estimation into RLHF. By combining quantile conformal prediction with heteroscedastic variance decomposition, the method enables robust quantification of reward confidence. This uncertainty estimate is then leveraged within Generalized Reward-Preference Optimization (GRPO) to dynamically reweight advantage computations, effectively downweighting unreliable signals. Experiments demonstrate that the approach significantly improves reward model calibration, reduces reward hacking, and enhances policy alignment across multiple benchmarks, including HelpSteer, UltraFeedback, and PKU-SafeRLHF.

Abstract

Reinforcement learning from human feedback (RLHF) aligns large language models by training reward models on preference data and optimizing policies to maximize predicted rewards. However, this pipeline faces two fundamental challenges: (1) reward models cannot signal when their predictions are unreliable, since they usually act as deterministic point estimators; and (2) modern group-based policy optimization can amplify unreliable reward signals, as exemplified by GRPO's uniform treatment of rewards during advantage computation. As policies explore increasingly diverse responses, these two limitations create a critical vulnerability: unreliable reward estimates may be granted disproportionate influence, triggering severe reward hacking. We propose Uncertainty-Aware Reward Modeling (UARM), which equips reward models with calibrated uncertainty via quantile-based conformal prediction and reweights GRPO advantages through heteroscedastic variance decomposition. Experiments across HelpSteer, UltraFeedback, and PKU-SafeRLHF demonstrate that UARM significantly improves reward model calibration, reduces reward hacking, and enhances downstream alignment quality compared to standard GRPO and uncertainty-agnostic baselines.

Problem

• reward hacking
• reward model uncertainty
• RLHF
• policy optimization
• reliability of reward signals

Innovation

• Uncertainty-Aware Reward Modeling
• Conformal Prediction
• Heteroscedastic Variance
• Reward Hacking
• GRPO