Problem
Research questions and friction points this paper is trying to address.
on-policy distillation
competence-aware
entropy-guided
supervision efficiency
knowledge distillation
SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision
🤖 AI Summary
This work addresses the inefficiency in on-policy distillation caused by ignoring the dynamic evolution of student model capabilities, which leads to redundant or noisy gradients at the token, training-stage, and prompt levels. To mitigate this, the authors propose SEAD, a novel framework that incorporates student capability awareness into on-policy distillation for the first time, using entropy as a unified metric to enable synergistic optimization across three granularities: skipping gradient updates for low-information tokens based on joint teacher–student entropy, dynamically reversing the direction of KL divergence via cosine scheduling, and introducing an easy-to-hard curriculum of prompts gated by student capability. Evaluated on OLMo-3 models (7B–32B), SEAD achieves an average accuracy gain of 4.8% over the baseline across six mathematical benchmarks, with ablation studies confirming the superadditive contributions of its components.
📝 Abstract
On-policy distillation (OPD) has a property absent in offline distillation and RL: teacher supervision quality depends on student competence. Incoherent rollouts yield noisy gradients; already-mastered tokens yield redundant ones. This creates waste at three scales (tokens, training phases, and prompts) yet existing methods supervise uniformly. We introduce SEAD, which uses entropy as a unified probe of this competence-dependent degradation at three scales: (1) joint teacher-student entropy partitions tokens into zones receiving tailored divergences or zero gradient (approx. 50% skipped); (2) a cosine schedule anneals from forward to reverse KL as competence grows; (3) a competence-gated curriculum introduces prompts easy-to-hard. These components are symbiotically necessary: token selection requires coherent rollouts (curriculum), annealing requires monotonic improvement (also curriculum). On OLMo-3 (7B to 32B), SEAD achieves +4.8 avg accuracy over vanilla OPD across six math benchmarks, with ablations confirming super-additive interactions.
Innovation
Methods, ideas, or system contributions that make the work stand out.
on-policy distillation
entropy-guided supervision
competence-aware learning
curriculum learning
adaptive KL annealing
🔎 Similar Papers
No similar papers found.
