Reinforcement Learning via Self-Distillation
This work addresses the challenge in reinforcement learning (RL) for verifiable domains such as code generation and mathematical reasoning, where reliance on sparse scalar rewards impedes effective credit assignment despite abundant textual feedback from the environment. To overcome this limitation, the authors propose Self-Distillation Policy Optimization (SDPO), a novel approach that leverages the model’s own predictions of corrective tokens from textual feedback to construct dense supervision signals. These signals are then distilled back into the policy network via a self-teaching mechanism, enabling context-aware self-correction without external teachers or explicit reward models. SDPO is the first method to formally treat textual feedback as an RL signal, significantly outperforming existing RLVR approaches on scientific reasoning, tool usage, and competitive programming tasks in LiveCodeBench v6. At inference time, applying SDPO to a single problem reduces the number of attempts needed to achieve comparable success rates by a factor of three.