DOPD: Dual On-policy Distillation

2026-06-29

AI Summary

This work addresses the “privileged hallucination” problem in policy distillation—where student models struggle to distinguish between transferable capability gaps and irreproducible information asymmetry caused by privileged information—by proposing an advantage-aware dual distillation paradigm. The method dynamically allocates token-level supervision signals based on the advantage discrepancy and relative probability between teacher and student policies, and introduces, for the first time, a dynamic routing mechanism that adaptively selects the supervision source per token between the teacher and the student itself. Combined with non-uniform supervision weighting and explicit modeling of privileged information, this approach significantly enhances distillation quality. Experiments demonstrate consistent superiority over standard policy distillation and other baselines across both large language models and vision-language models, with notable improvements in stability, robustness, continual learning, and out-of-distribution generalization.

Abstract

On-policy distillation (OPD) offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals. To furnish high-quality supervision sources and thereby elevate the performance frontier of distillation, an intuitive direction is to infuse privileged information to either teacher or student itself. However, this additional input induces a potential failure mode we dub privilege illusion: a pattern that conflates the transferable capability gap that students are meant to close, and the information asymmetry gap that can only be mimicked but never replicated. This issue is further amplified by the inherent non-uniformity of token-level supervision, where only a small subset of tokens carries pivotal capability-bearing signals. To this end, we propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies based on their advantage gap and relative probabilities. Each token receives supervision of different strength, objective, and strategy from either teacher or student itself, which transfers credible capability while simultaneously receiving auxiliary signals, to alleviate privilege illusion. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD and other counterparts. Further results on stability, robustness, continual learning, and out-of-distribution tasks validate its superiority.

Problem

• privilege illusion
• on-policy distillation
• token-level supervision
• information asymmetry
• capability transfer

Innovation

• On-policy Distillation
• Privilege Illusion
• Token-level Supervision
• Advantage-aware Routing
• Dual Distillation