Problem
Research questions and friction points this paper is trying to address.
on-policy distillation
probability transport
teacher-student disagreement
Distillation as Probability Transport: Routed On-Policy Distillation
🤖 AI Summary
该研究通过将在线策略蒸馏视为概率传输问题，并提出RouteOPD方法，以更精确地指导学生模型学习教师模型的知识，从而提高性能和减少背景泄漏。
📝 Abstract
On-policy distillation (OPD) transfers teacher knowledge on student-generated trajectories, but efficient sampled objectives reduce the teacher distribution to scalar credit on individual tokens. Such credit indicates whether a token should gain or lose probability, yet leaves the corresponding redistribution unspecified. We recast OPD as teacher-guided probability transport and propose RouteOPD (Routed On-Policy Distillation), which decomposes local teacher--student disagreement into student-excess sources and teacher-deficit destinations and couples them into explicit transport pairs. RouteOPD optimizes pairwise log-odds toward jointly realizable targets obtained from a bounded teacher potential, while adapting the transport budget to the concentration of teacher demand. This formulation directs updates toward teacher-preferred destinations and controls their magnitude within a single transport operator. Experiments across four teacher--student settings and four mathematical-reasoning benchmarks demonstrate that RouteOPD consistently outperforms sampled reverse-KL OPD, with improvements accompanied by higher routing fidelity and lower background leakage. These results demonstrate the effectiveness of explicitly modeling probability transport in on-policy distillation.
Innovation
Methods, ideas, or system contributions that make the work stand out.
probability transport
on-policy distillation
teacher-guided
🔎 Similar Papers
2024-07-21arXiv.orgCitations: 1
