Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards

2026-06-17

AI Summary

This work addresses the limitations of existing verifiable reward-based reinforcement learning methods, which uniformly distribute credit across all tokens and struggle to highlight critical reasoning steps, often relying on external models or ground-truth answers. The authors propose a self-conditioned credit assignment mechanism that operates purely within a reinforcement learning with verifiable rewards (RLVR) setting, leveraging only the model’s own verified trajectories. By employing per-token KL divergence as multiplicative gradient weights, the method enables fine-grained, supervision-free credit assignment. Integrated with the GRPO framework and inspired by self-distillation, the approach includes a theoretical proof that multi-trajectory self-teacher distillation is infeasible. Experiments demonstrate consistent improvements, outperforming GRPO by 8.1% and DAPO by 5.9% on average across five benchmarks—including mathematical reasoning, code generation, and agent tasks—and exhibiting superior out-of-distribution generalization compared to OPD.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in training LLMs for reasoning tasks, but representative methods such as GRPO assign uniform credit across all tokens, wasting gradient on routine tokens while under-crediting pivotal reasoning steps. Existing token-level credit assignment methods require resources beyond the model's own rollouts. GRPO variants rely on process reward models or ground-truth answers. Knowledge distillation assigns credit through per-token divergence but requires external teachers (On-Policy Distillation) or privileged information (On-Policy Self Distillation). However, these dependencies limit applicability in the pure RLVR setting. We observe that conditioning the model on its own verified trajectories induces a measurable per-token KL divergence between the original and conditioned distributions, and prove that distilling from a self-teacher constructed by verified trajectories leads to infeasible weighted-average solutions when multiple verified trajectories exist. We propose SC-GRPO (Self-Conditioned GRPO), which uses KL divergence mentioned before as a multiplicative weight on GRPO gradients. Across five benchmarks spanning math, code, and agentic tasks, SC-GRPO consistently outperforms 8.1% over GRPO and 5.9% over DAPO with stronger OOD performance. Moreover, SC-GRPO achieves higher performance than OPD.

Problem

• credit assignment
• reinforcement learning with verifiable rewards
• token-level credit
• self-conditioning
• RLVR

Innovation

• Self-Conditioned Credit Assignment
• KL Divergence Weighting
• Reinforcement Learning with Verifiable Rewards
• Token-level Gradient Modulation
• SC-GRPO