C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences

2026-04-15

AI Summary

Existing reward modeling approaches based on rubrics rely heavily on costly human annotations and are prone to being misled by low-quality rubrics, limiting their scalability. This work proposes C2, a framework that, for the first time, enables scalable rubric-augmented reward modeling without requiring external rubric annotations. C2 achieves this through a critical collaboration between a reward model and a rubric generator trained solely on binary preferences, synthesizing pairs of effective and misleading rubrics to jointly train both a generator and a verifier. During inference, the framework selects trustworthy rubrics to assist judgment. Experiments show that C2 improves performance by 6.5 points on RM-Bench and increases length-controlled win rates by 6.0 points on AlpacaEval 2.0; notably, its 8B reward model matches the performance of methods using rubrics generated by models four times larger.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Rubric-augmented verification guides reward models with explicit evaluation criteria, yielding more reliable judgments than single-model verification. However, most existing methods require costly rubric annotations, limiting scalability. Moreover, we find that rubric generation is vulnerable to a failure of cooperation; low-quality rubrics actively mislead reward models rather than help. Inspired by the principle of cooperative communication, we propose Cooperative yet Critical reward modeling (C2), a framework that significantly improves reward model judgments by having the reward model critically collaborate with a rubric generator trained solely from binary preferences. In C2, we synthesize helpful and misleading rubric pairs by measuring how each rubric shifts the reward model toward or away from the correct preference. Using these contrastive pairs, we train a cooperative rubric generator to propose helpful rubrics, and a critical verifier to assess rubric validity before making its judgment, following only rubrics it deems helpful at inference time. C2 outperforms reasoning reward models trained on the same binary preferences, with gains of up to 6.5 points on RM-Bench and 6.0 points length-controlled win rate on AlpacaEval 2.0. Without external rubric annotations, C2 enables an 8B reward model to match performance achieved with rubrics from a 4$\times$ larger model. Overall, our work demonstrates that eliciting deliberate cooperation in rubric-augmented verification makes reward models more trustworthy in a scalable way.

Problem

• rubric-augmented reward modeling
• binary preferences
• scalability
• cooperative communication
• reward model reliability

Innovation

• rubric-augmented reward modeling
• binary preferences
• cooperative communication
• contrastive rubric pairs
• critical verification