SR-OPSD: Self-Referenced On-Policy Self-Distillation

2026-08-10

AI Summary

This work addresses the instability and distributional collapse commonly observed in on-policy self-distillation (OPSD), where the co-evolution of teacher policy, student policy, and context distribution leads to optimization challenges. To mitigate this, we propose a self-referential distillation mechanism that integrates the self-teacher policy with a reference policy through a geometric interpolation objective derived from a variational characterization, evaluated within contexts generated by a fixed student policy. By generalizing projection geometry via the Rényi divergence family, our approach decouples distillation target localization from student policy projection. The interpolation coefficient governs the target location, while the Rényi order modulates sensitivity to token-level density ratios, substantially enhancing training stability. Empirically, the method achieves state-of-the-art or competitive performance across scientific evaluation, mathematical reasoning, and code generation tasks on multiple large language models.

Abstract

On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on trajectories generated by the policy to be optimized, providing a useful complement to reinforcement learning with sparse outcome rewards. However, the self-teacher policy used in OPSD is typically a stop-gradient or exponential-moving-average copy of the policy conditioned on additional context information, and thus co-evolves with both the student policy and its on-policy context distribution. Directly matching such a moving target with a fixed projection objective can lead to unstable optimization or excessive distributional concentration. This nature of OPSD motivates the proposed \emph{Self-Referenced On-Policy Self-Distillation (SR-OPSD)}. At fixed student-generated contexts, a token-level variational characterization identifies the effective distillation target as a geometric interpolation between the self-teacher policy and a reference policy. Meanwhile, we use the Rényi divergence family to generalize the projection geometry. This formulation separates \emph{where} the adaptive target is placed from \emph{how} the student is projected toward it: the interpolation coefficient controls underlying target, while the Rényi order controls the projection geometry and its sensitivity to token-level density ratios. Extensive experiments across scientific evaluation, mathematical reasoning, and coding generation tasks with multiple large language models show that SR-OPSD achieves the state-of-the-art or competitive performance across various settings.

Problem

• On-Policy Self-Distillation
• self-teacher policy
• optimization instability
• distributional concentration
• moving target

Innovation

• Self-Referenced On-Policy Self-Distillation
• Rényi divergence
• geometric interpolation
• token-level distillation
• policy optimization