Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation

2026-05-12

AI Summary

This work uncovers the parameter-level mechanisms underlying the efficiency of Online Policy Distillation (OPD), revealing its “foresight” during early training stages: it simultaneously allocates attention to critical reasoning modules and aligns with the low-rank subspace ultimately occupied by the optimized policy. Building on this insight, we propose EffOPD, a plug-and-play acceleration method that requires no additional trainable components or intricate hyperparameter tuning. Instead, EffOPD leverages module utility analysis, a metric for low-rank subspace alignment, and adaptive extrapolation step-size selection to enable highly efficient training. Experimental results demonstrate that EffOPD achieves an average 3× speedup over standard OPD while preserving comparable final performance.

Abstract

On-policy distillation (OPD) has emerged as an efficient post-training paradigm for large language models. However, existing studies largely attribute this advantage to denser and more stable supervision, while the parameter-level mechanisms underlying OPD's efficiency remain poorly understood. In this work, we argue that OPD's efficiency stems from a form of ``foresight'': it establishes a stable update trajectory toward the final model early in training. This foresight manifests in two aspects. First, at the \textbf{Module-Allocation Level}, OPD identifies regions with low marginal utility and concentrates updates on modules that are more critical to reasoning. Second, at the \textbf{Update-Direction Level}, OPD exhibits stronger low-rank concentration, with its dominant subspaces aligning closely with the final update subspace early in training. Building on these findings, we propose \textbf{EffOPD}, a plug-and-play acceleration method that speeds up OPD by adaptively selecting an extrapolation step size and moving along the current update direction. EffOPD requires no additional trainable modules or complex hyperparameter tuning, and achieves an average training acceleration of $3\times$ while maintaining comparable final performance. Overall, our findings provide a parameter-dynamics perspective for understanding the efficiency of OPD and offer practical insights for designing more efficient post-training methods for large language models.

Problem

• On-Policy Distillation
• parameter dynamics
• training efficiency
• large language models
• update trajectory

Innovation

• On-Policy Distillation
• Foresight Mechanism
• Low-Rank Concentration
• EffOPD
• Parameter Dynamics