Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?

2024-10-08

arXiv.org

AI Summary

Current reward modeling (RM) evaluation relies heavily on validation-set preference accuracy, yet this metric exhibits weak correlation with downstream reinforcement learning (RL) policy performance and thus fails to reliably predict optimization outcomes. Method: We construct a controlled synthetic environment and conduct systematic analyses—including correlation studies, error attribution, and theoretical modeling of Goodhart’s law—to investigate the validity of accuracy as a proxy for RM quality. Contribution/Results: We identify, for the first time, that RM accuracy is compromised by a “regressive Goodhart effect”: apparent improvements often stem from overfitting to annotation noise rather than enhanced true preference modeling. Experiments demonstrate that RMs with similar accuracy scores can yield substantially divergent RL policy performance, and that accuracy’s proxy validity critically depends on data distribution and annotation quality. Our findings challenge the accuracy-centric RM evaluation paradigm and provide both theoretical grounding and empirical evidence for developing more robust RM assessment frameworks.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Reward Models (RMs) are crucial for aligning language models with human preferences. Currently, the evaluation of RMs depends on measuring accuracy against a validation set of manually annotated preference data. Although this method is straightforward and widely adopted, the relationship between RM accuracy and downstream policy performance remains under-explored. In this work, we conduct experiments in a synthetic setting to investigate how differences in RM measured by accuracy translate into gaps in optimized policy performance. Our findings reveal that while there is a weak positive correlation between accuracy and downstream performance, policies optimized towards RMs with similar accuracy can exhibit quite different performance. Moreover, we discover that the way of measuring accuracy significantly impacts its ability to predict the final policy performance. Through the lens of the Regressional Goodhart effect, we recognize that accuracy, when used for measuring RM quality, can fail to fully capture the potential RM overoptimization. This underscores the inadequacy of relying solely on accuracy to reflect their impact on policy optimization.

Problem

• Evaluating Reward Models accuracy
• Relationship between RM accuracy and policy performance
• Impact of accuracy measurement on policy optimization

Innovation

• Investigates reward model accuracy impact.
• Explores policy performance correlation differences.
• Highlights accuracy measurement limitations effects.