Purified OPSD: On-Policy Self-Distillation Without Losing How to Think

2026-07-02

AI Summary

This work addresses a critical limitation of standard Output Policy Self-Distillation (OPSD) in long-chain reasoning: its tendency to impair a model’s reflective capacity due to teacher signals contaminated by reference answers that introduce task-irrelevant biases. The study is the first to identify and isolate this non-transferable component, proposing a purification distillation mechanism based on residual decomposition and pointwise mutual information (PMI). By constructing a reference-free teacher model, the method extracts transferable reasoning correction signals and generates a purified target distribution for distillation. Experiments across four long-chain reasoning models and two datasets demonstrate that the approach significantly outperforms both baseline methods and standard OPSD, while effectively preserving the model’s original cognitive behaviors.

Abstract

On-policy self-distillation (OPSD) has emerged as a promising paradigm for improving LLM reasoning, where a privileged teacher with access to reference solutions provides token-level supervision on the student's own generated trajectories. However, we find that OPSD consistently fails on long chain-of-thought (long-CoT) reasoning models, yielding at best marginal gains while destabilizing the reflective reasoning capability these models depend on. Through a novel decomposition of the teacher's supervision signal, we identify the root cause: the teacher's supervision is dominated by a reference-induced component that drives rote memorization of reference-specific shortcuts, while the question-conditioned, inference-transferable component is ignored or actively opposed. Based on this diagnosis, we propose a two-step solution. First, we construct a reference-only teacher (the same model conditioned on the reference without the question) to isolate the non-transferable component of the supervision signal; the residual after subtracting this component captures the question-conditioned, inference-transferable correction. Second, we use pointwise mutual information (PMI) as the mechanism to transform this residual into a well-formed PMI target distribution that the student can directly distill from, filtering out the reference-induced shortcut. Experiments on four long-CoT models across two datasets demonstrate consistent improvements over both the base model and standard OPSD, while preserving the models' natural epistemic behavior throughout training.

Problem

• On-Policy Self-Distillation
• long chain-of-thought
• reasoning
• reference-induced shortcut
• reflective reasoning

Innovation

• on-policy self-distillation
• long chain-of-thought reasoning
• pointwise mutual information
• reference-induced shortcut
• inference-transferable supervision