Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling

2026-07-02

AI Summary

Existing unsupervised scoring rule generation methods rely on a single evaluation persona, often overlooking critical dimensions of human preference and thereby introducing blind spots in assessment. This work proposes a Multi-Roles Scoring Rule Generation framework (MRRG), which introduces, for the first time, a training- and reference-free collaborative multi-role mechanism. By jointly generating and fusing interpretable scoring rules from complementary personas, MRRG enables verifiable pairwise preference validation and provides reward signals for reinforcement learning. Integrating multi-role prompting, rule fusion, and verifiable reward modeling, the method supports GRPO-style reinforcement learning. Empirical results demonstrate that MRRG significantly outperforms single-role baselines across multiple preference validation benchmarks, yielding more comprehensive and reliable reward signals that effectively enhance open-domain generation quality.

Abstract

Reliable reward and preference signals are critical for evaluating and optimizing large language models on open-ended tasks. Rubric-based judges offer a transparent way to decompose such judgments into explicit evaluation criteria, but existing annotation-free rubric generators typically rely on a single generic evaluator. As a result, they may overlook important dimensions of human preference, a failure mode we term dimensional blind spots. To address this limitation, we propose Multi-Role Rubric Generation (MRRG), a training-free and reference-free framework that elicits evaluation criteria from multiple complementary roles and consolidates them into an auditable rubric-based scorer. This scorer can be used both to validate pairwise preferences and to provide rewards for GRPO-style Reinforcement Learning with Verifiable Rewards (RLVR). Experiments on preference validation benchmarks show that MRRG consistently outperforms single-role rubric generation baselines across multiple backbone models. Further RLVR experiments demonstrate that MRRG yields a stronger reward signal for improving open-ended generation.

Problem

• rubric generation
• dimensional blind spots
• LLM judging
• reward modeling
• preference evaluation

Innovation

• Multi-Role Rubric Generation
• Reward Modeling
• Preference Validation
• Reinforcement Learning with Verifiable Rewards
• Dimensional Blind Spots