DemoPSD: Disagreement-Modulated Policy Self-Distillation

2026-07-02

AI Summary

This work addresses the limitations of large language models in policy self-distillation, where reliance on privileged information from the teacher model often leads to overfitting, restricted exploration, degraded cross-domain generalization, and information leakage. To mitigate these issues, the authors propose DemoPSD, a novel framework that introduces, for the first time, an adaptive fusion mechanism based on the distributional discrepancy between student and teacher models. This mechanism dynamically modulates the weight of teacher guidance at each token position and incorporates a reverse KL centroid objective to balance imitation learning with autonomous reasoning. Theoretical analysis demonstrates that DemoPSD effectively alleviates privileged information leakage while preserving exploration capacity. Empirical results show that DemoPSD significantly outperforms GRPO and SDPO across four scientific domains in SciKnowEval, achieves higher training entropy, and exhibits superior out-of-distribution generalization on the GPQA benchmark.

Abstract

On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization, while also introducing a more fundamental issue: *privileged information leakage*, where the student encodes answer-dependent shortcuts that are unavailable at test time. We introduce **DemoPSD**, a novel framework that resolves such problems through the idea of *selective adoption of teacher guidance*. Instead of fitting the full teacher distribution, DemoPSD steers the student toward a *reverse-KL barycenter target*, a weighted geometric combination of the teacher and student distributions, that naturally balances learning from the teacher with preserving the student's own reasoning capacity. We measure the difference between their distributions and use such a discrepancy to adaptively control the blending at each token position. We provably show that DemoPSD achieves **(1)** *leakage attenuation*, i.e., effective mitigation of privileged information leakage; and **(2)** *exploration preservation*, i.e., preservation of exploration capacity under dense token-level distillation. Extensive experiments on SciKnowEval across four scientific fields show that DemoPSD outperforms both GRPO and SDPO while maintaining higher training entropy and robustly generalizing to out-of-distribution GPQA benchmarks.

Problem

• privileged information leakage
• on-policy self-distillation
• cross-domain generalization
• overfitting
• exploration suppression

Innovation

• self-distillation
• privileged information leakage
• reverse-KL barycenter
• exploration preservation
• on-policy learning