Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning

2026-06-13

AI Summary

This work addresses the challenge of fine-grained, token-level credit assignment in short-answer tasks, where scalar-reward-based reinforcement learning struggles to provide precise supervision. To overcome this limitation, the authors propose Hindsight Self-Distillation (HSD), a method that leverages successful reasoning trajectories within the same batch as conditional teacher signals. By aligning failed trajectories with accurate continuations starting precisely at path divergence points, HSD generates dense and spatially localized credit supervision without requiring additional sampling. Integrating path-conditioned self-distillation with token-level guidance, the approach significantly outperforms existing GRPO variants and self-distillation baselines on Qwen3-8B and Qwen3-32B models, achieving particularly strong gains on mathematical and code-related short-answer benchmarks such as AIME.

Abstract

Reinforcement learning from verifiable rewards assigns a single scalar to each rollout, leaving token-level credit assignment underspecified in long reasoning traces. On-policy self-distillation addresses this by letting the same model act as a teacher conditioned on privileged information, producing a dense per-token signal. But the common choice of a ground-truth answer is only an endpoint cue: on terse-answer tasks, the teacher falls silent at the intermediate positions where path-level guidance matters most. We propose Hindsight Self-Distillation (HSD), which conditions the teacher on a successful peer rollout drawn from the current training group. Such a peer is an exact sample from the success-conditioned policy, requiring no additional sampled rollouts. By providing a full successful continuation rather than only the final answer, the resulting credit signal concentrates at the divergence position between a failed rollout and a successful peer. Across Qwen3-8B and Qwen3-32B on math and code benchmarks, HSD obtains the best result against GRPO variants and on-policy distillation baselines, with the largest gains on terse-answer tasks such as AIME.

Problem

• credit assignment
• self-distillation
• reasoning traces
• reinforcement learning
• large language models

Innovation

• Hindsight Self-Distillation
• credit assignment
• path-conditioned distillation
• reinforcement learning
• large language models