Step-wise Rubric Rewards for LLM Reasoning

2026-05-17

AI Summary

This work addresses the limitation of existing reinforcement learning approaches that reward only the correctness of final answers, thereby lacking fine-grained supervision over intermediate reasoning steps and often leading to misaligned rewards and self-correction loops. To overcome this, the authors propose the Stepwise Rewarding by Rubrics (SRaR) framework, which introduces, for the first time, a step-level reward mechanism grounded in multidimensional scoring rubrics. Large language models serve as judges to attribute rubric criteria to specific reasoning steps, and a decoupled advantage estimator integrates step-wise and outcome-based rewards via cross-trajectory normalization, mitigating uniform supervision and reward hacking. A novel contrastive rubric dataset with 16K problems is constructed to support this approach. Experiments demonstrate that SRaR significantly outperforms RaR across six mathematical reasoning benchmarks, improving average accuracy by 3.57% and 2.75% for Qwen3-8B and Qwen3-32B, respectively, increasing faithful reasoning on AIME 2025 from 34.5% to 46.7%, and reducing self-correction loops from 48.1% to 26.5%.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models, but rewards only final-answer correctness with no supervision over intermediate steps. Rubric-based methods such as Rubrics as Rewards (RaR) introduce finer-grained supervision by scoring rollouts against structured criteria, yet the rubric scores are still aggregated into a single scalar applied to the entire response, causing three weaknesses: loss of multi-criterion structure, uniform supervision of correct and incorrect steps, and reward hacking through unbounded self-correction. On 1,000 problems, we find 18.2% of steps in correct-answer responses are wrong yet positively rewarded, while 49.9% of steps in incorrect-answer responses are correct yet penalized. We introduce Step-wise Rubrics as Rewards (SRaR), an RLVR framework that (i) uses an LLM judge to attribute each rubric item to a specific reasoning step, (ii) normalizes per-step rubric scores across rollouts so only steps whose quality varies produce a learning signal, and (iii) combines the per-step reward with the outcome reward through a decoupled advantage estimator that keeps the outcome baseline stable. We further build a 16K-problem rubric dataset by contrastively distilling rubric items from correct and flawed reasoning paths sampled from a strong model. Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points on Qwen3-8B and 2.75 points on Qwen3-32B, raises the Faithful Reasoning Rate on AIME 2025 from 34.5% to 46.7%, and reduces self-correction looping from 48.1% to 26.5%.

Problem

• reinforcement learning
• reasoning supervision
• rubric-based rewards
• step-wise evaluation
• reward hacking

Innovation

• Step-wise Rubrics
• Reinforcement Learning with Verifiable Rewards
• Faithful Reasoning
• Decoupled Advantage Estimator
• Rubric-based Supervision