Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing

2026-05-07

AI Summary

This work addresses the limitations of standard knowledge distillation for autoregressive models, which suffers from distribution mismatch, and the inefficiency of existing on-policy approaches that rely on costly reinforcement learning frameworks. To overcome these challenges, the authors propose Near-Policy Distillation (NPD), the first method to combine asynchronous generation with selective sample filtering while preserving near-policy characteristics. NPD achieves highly efficient supervised fine-tuning through several key innovations: decoupling student model inference from training, sequence packing, sparse updates, and a Δ-IFD-based sample filtering mechanism. Experimental results demonstrate that NPD accelerates training by 8.1× compared to on-policy baselines and outperforms standard supervised fine-tuning by 8.09% in performance. Notably, it enables the smaller openPangu-Embedded-1B model to surpass the larger Qwen3-1.7B, achieving a state-of-the-art score of 68.73%.

Abstract

Standard knowledge distillation for autoregressive models often suffers from distribution mismatch. While on-policy methods mitigate this by leveraging student-generated outputs, they rely on computationally expensive Reinforcement Learning (RL) frameworks. To improve efficiency, we propose Near-Policy Distillation (NPD), an asynchronous approach that decouples student generation from training. This reformulation enables Supervised Fine-Tuning (SFT) with sequence packing. However, asynchronous updates inevitably introduce policy lag and sample noise, which can cause the behavior to drift from near-policy toward off-policy. To counteract this without sacrificing efficiency, NPD integrates sparse student updates and the $Δ$-IFD filtering mechanism, a heuristic sample selection mechanism that empirically stabilizes the optimization trajectory. By filtering extreme out-of-distribution samples, $Δ$-IFD prevents noise from dominating the gradients, ensuring updates remain within a safe proximal learning zone. Empirically, the NPD framework achieves a 8.1x speedup over on-policy baselines and outperforms SFT by 8.09%. Crucially, by effectively narrowing the exploration space for subsequent RL, our method enables openPangu-Embedded-1B to reach a state-of-the-art score of 68.73%, outperforming the substantially larger Qwen3-1.7B. Codes will be released soon.

Problem

• knowledge distillation
• distribution mismatch
• on-policy learning
• autoregressive models
• computational efficiency

Innovation

• Near-Policy Distillation
• Asynchronous Generation
• Δ-IFD Filtering
• Sequence Packing
• Policy Lag Mitigation