PEBS: Per-rater Empirical-Bayes Shrinkage for RLHF Reward-Model Calibration

2026-06-25

AI Summary

This work addresses the limitation of existing reinforcement learning from human feedback (RLHF) reward models, which employ a single global affine calibrator and thus fail to capture systematic differences in annotators’ scoring scales, leading to miscalibrated outputs that deviate from the true annotation distribution. To resolve this, the authors propose an annotator-granular posterior calibration method that estimates individual affine parameters for each annotator without modifying the original reward model. Crucially, they introduce, for the first time, a Morris–James–Stein empirical Bayes shrinkage mechanism, enabling a closed-form solution without requiring model retraining. The approach substantially improves calibration generalization, reducing RMSE by 8.58% on the within-user held-out test set of the PRISM dataset and achieving a further 9.66% reduction on the PluriHarms harmfulness scoring task.

Abstract

Reward models for Reinforcement Learning from Human Feedback (RLHF) pool preferences across thousands of annotators and fit one global affine calibrator, collapsing raters with systematically different rating-scale offsets and slopes into a single average-rater fit that does not match any individual annotator. PEBS is a per-rater empirical-Bayes shrinkage estimator: it fits per-rater affine calibrators on a held-out slice of each annotator's ratings and applies Morris-James-Stein empirical-Bayes shrinkage toward the population mean, in closed form and without retraining the reward model. On PRISM, PEBS reduces within-user held-out RMSE by 8.58% over the pooled population-slope baseline. The procedure replicates on PluriHarms harm ratings (Qwen-2.5 base, in-family) with a +9.66% RMSE reduction over the same population-slope baseline. PEBS is a closed-form post-hoc estimator for annotator-specific affine calibration in RLHF reward modeling; it leaves the reward base model unchanged and estimates only the rater-level map used at inference time for new ratings.

Problem

• RLHF
• reward model calibration
• annotator heterogeneity
• affine calibration
• rating-scale offsets

Innovation

• empirical-Bayes shrinkage
• per-rater calibration
• RLHF reward modeling
• affine calibration
• closed-form estimator