EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning

2026-06-22

AI Summary

Static rubrics in reinforcement learning for large language models struggle to adapt to policy evolution, often leading to reward saturation and model exploitation. To address this, this work proposes EvoRubrics, a framework that enables fine-grained, real-time adversarial co-evolution between the policy model and the scoring rubric. At each training step, the policy and a rubric generator interact dynamically to produce effective reward signals without external supervision or reliance on state-of-the-art models, implicitly constructing a curriculum in the process. The approach substantially outperforms existing static and dynamic rubric-based methods, achieving significant performance gains across multiple benchmark tasks. Moreover, the trained rubric generator functions as a generalizable reward model that can be transferred to other settings.

Abstract

Rubric-based rewards offer interpretable and fine-grained optimization signals for reinforcement learning in open-ended tasks where verifiable answers are unavailable. However, pre-constructed rubrics remain static throughout training, creating a fundamental mismatch with the evolving policy: fixed criteria gradually lose discriminative power as the model improves, leading to reward saturation and potential hacking. Recent dynamic rubric methods partially address this but rely on external frontier models or ground-truth answers, and update rubrics only at coarse granularity. We propose EvoRubrics, a co-evolutionary RL framework where a Policy LLM and a Rubric Generator jointly improve through adversarial interaction within each training step. As the policy improves under the rubric generator's guidance, the rubric generator adapts its criteria to remain discriminative and informative, enabling evaluation to track the policy in real time and naturally inducing an automatic curriculum. Experiments show that EvoRubrics consistently outperforms static and dynamic rubric baselines across benchmarks. The learned Rubric Generator further generalizes as a transferable reward model. Notably, even a fully self-supervised variant without any external supervision achieves meaningful gains, suggesting that co-evolution between generation and evaluation alone can provide sufficiently rich learning signals. Our code is publicly available at https://anonymous.4open.science/r/EvoRubrics-2155/.

Problem

• dynamic rubrics
• reinforcement learning
• large language models
• reward saturation
• open-ended tasks

Innovation

• adversarial co-evolution
• dynamic rubrics
• LLM reinforcement learning
• self-supervised reward
• automatic curriculum