Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation

2026-08-10

AI Summary

This work investigates the true source of performance gains in On-Policy Self-Distillation (OPSD): whether they stem from the teacher model’s privileged access to ground-truth solutions or from contextual shifts induced by these solutions that alter teacher behavior. To disentangle these factors, we propose OP²SD, a novel method that substitutes the teacher’s input with solutions from other problems while preserving the student’s trajectory and distillation target. Experiments across three large language models and three mathematical reasoning benchmarks demonstrate that OP²SD not only significantly outperforms baseline approaches but also matches the performance of standard OPSD. These findings reveal that context-induced changes in teacher behavior—not privileged information—are the key driver of improvement, thereby challenging conventional interpretations of the OPSD mechanism.

Abstract

On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory. However, this interpretation conflates two effects. The reference solution not only reveals the answer to the current instance but also changes the context under which the teacher provides token-level supervision. We investigate the role of target-specific privilege with $\mathrm{OP}^{2}\mathrm{SD}$ (On-Policy Self-Distillation from Other Problems), which replaces the paired reference with a problem and solution from a different example, while preserving the student rollout, teacher, and distillation objective. Across three models and three mathematics benchmarks, $\mathrm{OP}^{2}\mathrm{SD}$ improves over the base model, remains competitive with OPSD. The success of $\mathrm{OP}^{2}\mathrm{SD}$ implies that OPSD gains do not necessarily come from access to the reference solution, and that the teacher's context-induced behavior is an important factor.

Problem

• On-Policy Self-Distillation
• privileged information
• context-induced behavior
• teacher supervision
• distillation

Innovation

• On-Policy Self-Distillation
• Context-Induced Behavior
• Privileged Information
• OP2SD
• Token-Level Supervision