Mismatch Matters: On-Policy Distillation Beyond Token Agreement

2026-08-10

AI Summary

This work addresses a critical degradation issue in online policy distillation (OPD), where student models often fall into repetitive loops, leading to superficial token-level alignment with the teacher but poor overall response quality. The study presents the first systematic analysis of teacher-student distributional mismatch, identifying two key failure modes: redundant token generation and missing probability mass. To mitigate these issues, the authors propose TIDE, a novel distillation framework that employs bounded Hellinger distance shaping to suppress redundancy and injects an analytical top-K prior from the teacher to recover missing probability mass—without requiring additional sampling. Evaluated on mathematical reasoning benchmarks using Qwen3 teacher-student pairs, TIDE substantially outperforms standard OPD and other baselines, improving Avg@8 from 6.9% to 20.3% under severe mismatch conditions, reducing response length by 3.6×, and significantly decreasing formatting errors.

Abstract

On-policy distillation (OPD) has emerged as a core component of modern LLM post-training pipelines, yet we reveal a failure mode: degenerate agreement, where students exploit repetitive loops to achieve near-perfect token agreement with the teacher despite globally flawed responses. We therefore shift our focus from agreement to teacher-student mismatch, and find that mismatch tokens can be mainly categorized into two types: student-excess tokens and student-deficit tokens. Student-excess tokens are generated by the student but assigned near-zero probability by the teacher; their log-ratio corrections grow unbounded and destabilize the update. Student-deficit tokens, in contrast, are preferred by the teacher but rarely sampled by the student; their absence blocks the transfer of the teacher's reasoning patterns. To tackle these mismatch directions, we propose TIDE (Token-level Independent Deficit-Excess correction), which applies bounded Hellinger shaping to suppress the most severe sampled excesses and an analytic teacher top-$K$ injection to restore deficient probability mass without requiring deficit tokens to be sampled. Across mathematical reasoning benchmarks with multiple Qwen3 teacher-student pairs, TIDE consistently outperforms standard OPD and recent token-selection and reward-shaping baselines. Moreover, the gains of TIDE are more pronounced under strong teacher-student mismatch, where it improves Avg@8 from 6.9% to 20.3%, reduces average response length by a factor of 3.6, and substantially reduces formatting failures. Code is available at https://github.com/yzc-666/TIDE

Problem

• on-policy distillation
• teacher-student mismatch
• student-excess tokens
• student-deficit tokens
• token agreement

Innovation

• on-policy distillation
• teacher-student mismatch
• TIDE
• Hellinger shaping
• top-K injection