SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation

2026-05-08

AI Summary

This work addresses the limitation of conventional online policy distillation under heterogeneous tokenizers, where enforcing strict token-by-token alignment discards substantial teacher supervision signals at vocabulary mismatches. To overcome this, the authors propose SimCT, which introduces the finest-grained jointly segmentable units as the supervisory interface for cross-tokenizer distillation and expands the alignment space by leveraging short multi-token continuation sequences generatable by both teacher and student. This approach preserves the original distillation loss formulation while recovering previously discarded supervision signals. Evaluated across three heterogeneous teacher-student model pairs on mathematical reasoning and code generation tasks, SimCT consistently outperforms both shared-vocabulary distillation and existing cross-tokenizer baselines, achieving significant performance gains.

Abstract

On-policy distillation (OPD) is a standard tool for transferring teacher behavior to a smaller student, but it implicitly assumes that teacher and student predictions are comparable token by token, an assumption that fails whenever the two models tokenize the same text differently. Under heterogeneous tokenizers, exact shared-token matching silently discards a large fraction of the teacher signal at precisely the positions where vocabularies disagree. We propose \textbf{\underline{Sim}ple \underline{C}ross-\underline{T}okenizer OPD (SimCT)}, which restores this signal by enlarging the supervision space: alongside shared tokens, SimCT compares teacher and student over short multi-token continuations that both tokenizers can realize, leaving the OPD loss form itself unchanged. We show that these units are the finest jointly tokenizable supervision interface, and that coarser alternatives remove teacher-student distinctions that are useful for on-policy learning. Across three heterogeneous teacher-student pairs on mathematical reasoning and code-generation benchmarks, SimCT shows consistent gains over shared-vocabulary OPD and representative cross-tokenizer baselines, with ablations confirming that the improvements come from recovering supervision discarded by exact shared-token matching. Code is available at \href{https://github.com/sunjie279/SimCT-}{https://github.com/sunjie279/SimCT-}.

Problem

• on-policy distillation
• cross-tokenizer
• heterogeneous tokenizers
• supervision loss
• tokenization mismatch

Innovation

• on-policy distillation
• cross-tokenizer
• knowledge distillation
• heterogeneous tokenizers
• supervision recovery