reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs

2025-03-14

AI Summary

This work addresses the insufficient robustness of reward models under semantically preserving input transformations. To this end, the authors propose the first systematic robustness evaluation framework specifically designed for reward modeling. They introduce reWordBench—a benchmark encompassing synonym substitution, syntactic rephrasing, and other semantic- and ranking-preserving perturbations—revealing substantial performance degradation (e.g., ~50% drop on the Chat Hard subset) attributable to overfitting in prevailing models. Methodologically, they pioneer a synonym-scoring consistency objective and integrate contrastive robust training with transfer evaluation on RewardBench subsets. Experimental results demonstrate significantly improved generalization across diverse transformations. In end-to-end alignment-stage validation, the proposed approach outperforms standard reward models on 59% of comparative tasks, yielding higher-quality model outputs.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Reward models have become a staple in modern NLP, serving as not only a scalable text evaluator, but also an indispensable component in many alignment recipes and inference-time algorithms. However, while recent reward models increase performance on standard benchmarks, this may partly be due to overfitting effects, which would confound an understanding of their true capability. In this work, we scrutinize the robustness of reward models and the extent of such overfitting. We build **reWordBench**, which systematically transforms reward model inputs in meaning- or ranking-preserving ways. We show that state-of-the-art reward models suffer from substantial performance degradation even with minor input transformations, sometimes dropping to significantly below-random accuracy, suggesting brittleness. To improve reward model robustness, we propose to explicitly train them to assign similar scores to paraphrases, and find that this approach also improves robustness to other distinct kinds of transformations. For example, our robust reward model reduces such degradation by roughly half for the Chat Hard subset in RewardBench. Furthermore, when used in alignment, our robust reward models demonstrate better utility and lead to higher-quality outputs, winning in up to 59% of instances against a standardly trained RM.

Problem

• Assessing reward model robustness to input transformations
• Identifying overfitting effects in reward model performance
• Improving reward model robustness through paraphrase training

Innovation

• Developed reWordBench for testing reward model robustness
• Proposed training reward models with paraphrase similarity
• Enhanced reward model utility in alignment tasks