X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation

2026-05-20

AI Summary

This work addresses the challenges of rare token inefficacy and overly conservative matching in cross-tokenizer knowledge distillation, which arise from vocabulary incompatibility. To overcome these issues, the authors propose X-Token, a method that introduces a sparse projection matrix combined with partition-free alignment and relaxed matching strategies. It innovatively integrates P-KL and H-KL loss functions to dynamically handle critical token misalignment and approximate token equivalence. The projection matrix is initialized via string-based rules, and a cross-tokenizer logits alignment mechanism enables effective multi-teacher distillation. Evaluated on Llama-3.2-1B, X-Token outperforms the state-of-the-art GOLD method by an average of 3.82 points with a Qwen3-4B teacher and 0.5 points with a Phi-4-Mini teacher; performance further improves by 1.3 points in a dual-teacher setting.

Abstract

Cross-tokenizer knowledge distillation allows a student model to learn from teachers with incompatible vocabularies. Prior work operates on hidden states or logits; the latter is preferred as a drop-in replacement requiring no auxiliary components. Logit-based methods either use only the correct-token probability, missing the full 'dark knowledge' in the teacher's distribution, or operate on the full output distribution, relying on strict token partitioning and/or unprincipled heuristic ranking. We identify two key shortcomings of full-distribution, logit-based methods: (i) an uncommon-token failure, where critical tokens fall into the unmatched subset (e.g., Llama's 1100 multi-digit numerals under digit-splitting Qwen supervision) and are suppressed during training, reducing GSM8k from 12.89 to 2.56 compared to same-tokenizer KD from a weaker teacher; and (ii) over-conservative matching, where strict 1-to-1 matching excludes near-equivalent tokens across surface forms. These failures require distinct remedies: eliminating the partition when critical tokens are misaligned, and refining it when alignment is reliable. We propose X-Token, an approach with two complementary loss formulations targeting these issues. P-KL removes partitioning and aligns the student's distribution with the teacher's via a sparse projection matrix W (initialized from tokenizer-level string rules) to address the uncommon-token failure. H-KL retains the hybrid form while relaxing matching to align each student token with its top-ranked teacher mapping under W. Both objectives share W and extend naturally to multiple teachers. Empirically, on Llama-3.2-1B, X-Token outperforms the current state of the art GOLD by +3.82 average points with a Qwen3-4B teacher and by +0.5 with a Phi-4-Mini teacher. Further, a two-teacher setup (Phi-4-mini + Llama-3B) improves over single-teacher distillation by +1.3 points.

Problem

• cross-tokenizer knowledge distillation
• logit-based distillation
• uncommon-token failure
• over-conservative matching
• vocabulary incompatibility

Innovation

• cross-tokenizer knowledge distillation
• projection-guided alignment
• sparse projection matrix
• logit-based distillation
• multi-teacher distillation