Problem
Research questions and friction points this paper is trying to address.
credit assignment
self-distillation
outcome verification
token-level scoring
on-policy learning
Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation
🤖 AI Summary
In outcome-verifier-based reinforcement learning, intermediate tokens lack a reliable credit assignment mechanism, and directly using likelihood shifts from privileged self-distillation as credit signals is prone to misleading outcomes. This work formally distinguishes token likelihood changes from genuine credit signals for the first time and introduces a three-dimensional evaluation framework encompassing action quality, feedback construction consistency, and alignment with training objectives. A decoupled verification methodology is developed to ensure sound credit assignment. Empirical experiments on a 20B-parameter model—employing policy self-distillation, post-hoc re-scoring, and paired comparisons—reveal that additive scoring performs near-randomly (AUC=0.505) and, after length correction, even favors incorrect reasoning. In contrast, the outcome-controlled baseline achieves 64.2% accuracy, significantly outperforming five token-scoring variants (24.2%–33.9%).
📝 Abstract
Outcome verifiers score completed reasoning traces but do not assign credit to intermediate tokens. Privileged self-distillation attempts to fill this gap by rescoring a model's own rollout with training-only information. A token likelihood change, however, is not automatically outcome credit. We separate three questions: whether the score tracks better actions, whether feedback construction changes what is compared, and what behavior the training loss reinforces. We establish these distinctions formally. When a rollout is scored using hindsight feedback written about that same rollout, its content determines both the tokens and the scoring context, creating direct self-dependence. Using feedback from another rollout of the same problem removes this dependence but does not guarantee a useful score. In matched experiments with a 20B model on AIME 2025, the implemented additive score is near chance (AUC=0.505) and slightly favors incorrect traces after length adjustment. In the paired comparison, the outcome-only control records 64.2\%, versus 24.2\%--33.9\% for five token-score variants. The results motivate validating score meaning, feedback construction, and training behavior separately before calling a likelihood signal credit.
Innovation
Methods, ideas, or system contributions that make the work stand out.
privileged self-distillation
token credit assignment
hindsight feedback
on-policy distillation
reasoning trace evaluation
🔎 Similar Papers
No similar papers found.
