Problem
Research questions and friction points this paper is trying to address.
on-policy distillation
temporal credit assignment
optimization stability
objective fidelity
variance
Beyond Token-Local Imitation: Reward-Compatible Temporal Credit Assignment for On-Policy Distillation
🤖 AI Summary
本文针对在线策略蒸馏中目标保真度与优化稳定性之间的权衡问题，提出了一种基于折扣时间信用分配的方法γOPD来平衡长期监督和优化稳定性。
📝 Abstract
On-policy distillation (OPD) has emerged as an effective approach for large language model post-training, yet existing objectives face a trade-off between objective fidelity and optimization stability. Token-level OPD provides stable but local supervision, whereas sequence-level OPD captures future credit at the cost of horizon-dependent variance. We establish a unified temporal-credit view of these formulations, showing that practical token-level OPD can be interpreted as a temporal approximation to the sequence-level reverse-KL gradient. Building on this connection, we propose $γ$OPD, which uses discounted temporal credit assignment to balance long-horizon supervision and optimization stability, while admitting a horizon-independent variance bound. We further develop a reward-compatible bounded mixing (RBM) mechanism for $γ\mathrm{OPD}$ that balances verifiable outcome feedback with the discounted OPD advantage to move beyond purely teacher-dependent optimization. Experiments on mathematical and code reasoning demonstrate consistent improvements over existing OPD methods across vanilla, size-mismatched, and multi-teacher distillation settings.
Innovation
Methods, ideas, or system contributions that make the work stand out.
discounted temporal credit assignment
reward-compatible bounded mixing (RBM)
optimization stability
🔎 Similar Papers
No similar papers found.
