Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation

2026-06-17

AI Summary

This work addresses the limitations of existing post-training methods for reasoning language models, which rely either on costly and noisy chain-of-thought annotations or on scalar rewards that lack fine-grained guidance. To overcome these challenges, the authors propose the first rubric-conditioned self-distillation framework that integrates task-specific, structured scoring rubrics into the training process, thereby providing token-level supervision signals for the reasoning trajectories generated by the student model and moving beyond the constraint of a single reference path. The approach employs a two-stage pipeline: first generating task-specific rubrics, then training the reasoning model via on-policy self-distillation augmented with fine-grained feedback derived from these rubrics. Evaluated across multiple scientific reasoning benchmarks, the method achieves consistent improvements, outperforming GRPO by 1.0 point and OPSD by 0.9 point on average, demonstrating a significant enhancement in reasoning capabilities.

Abstract

Post-training of reasoning language models is commonly driven by supervised distillation and reinforcement learning with verifiable rewards. Distillation often relies on chain-of-thought annotations that are expensive to obtain and may themselves be noisy, incomplete, or partially incorrect; even when the final solution is correct, an imperfect rationale can interfere with learning. Reinforcement learning with verified rewards, on the other hand, typically compresses evaluative feedback into a scalar signal, obscuring which aspects of a response should be improved. We propose \textbf{Rubric-Conditioned Self-Distillation}, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation. Our method conditions the teacher model on criterion-level rubrics and uses it to provide token-level guidance on the student's own sampled trajectories. This design avoids treating a single reference rationale as the sole supervision target. Instead, rubrics specify what a strong response should satisfy, enabling more fine-grained credit assignment over the reasoning process than scalar reward optimization. We instantiate this framework with a two-stage pipeline that first learns to generate task-specific rubrics and then trains a rubric-guided reasoner. We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance over the reasoning process, surpassing GRPO by 1.0 points and OPSD by 0.9 points on average.

Problem

• reward supervision
• reasoning language models
• chain-of-thought
• reinforcement learning
• fine-grained feedback

Innovation

• Rubric-Conditioned Self-Distillation
• fine-grained feedback
• reasoning language models
• token-level guidance
• structured evaluation rubrics