DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillation

2026-02-25

AI Summary

Existing cross-tokenizer knowledge distillation methods are limited by insufficient alignment at both sequence and lexical levels. This work proposes the DWA-KD framework, which introduces a novel dual-space entropy-weighting mechanism to prioritize information-rich tokens, thereby enhancing token-level distillation. For the first time, Soft-DTW is applied to both the embedding and final hidden layers to achieve simultaneous lexical and semantic sequence alignment across different tokenizers. By integrating dual-space KL divergence distillation, entropy-driven weighting, and Soft Dynamic Time Warping, the proposed method consistently outperforms state-of-the-art approaches on multiple NLP benchmarks. Ablation studies further confirm the effectiveness and complementarity of each component in the framework.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Knowledge Distillation (KD) has emerged as a crucial technique for compressing Large Language Models (LLMs). Although existing cross-tokenizer KD methods have made notable progress, their effectiveness remains constrained by suboptimal alignment across sequence and vocabulary levels. To address these limitations, we introduce Dual-Space Weighting and Time-Warped Alignment (DWA-KD), a novel cross-tokenizer distillation framework that enhances token-wise distillation through dual-space entropy-based weighting and achieves precise sequence-level alignment by leveraging both lexical and semantic information. At the token level, DWA-KD maps teacher representations into the student space and vice versa, performing dual-space KD via Kullback-Leibler divergence (KL). The process is modulated by dual-space weights that up-weight tokens where the student is uncertain and the teacher is confident, thereby focusing learning on informative tokens rather than treating all positions equally. At the sequence level, DWA-KD applies Soft Dynamic Time Warping (Soft-DTW) to both the embedding and final hidden-state layers, enabling robust alignment of lexical and contextual semantics between teacher and student sequences. Extensive experiments across diverse NLP benchmarks demonstrate that DWA-KD outperforms state-of-the-art KD baselines, while ablation studies confirm the complementary contributions of entropy-based token weighting and embedding and final hidden state layer Soft-DTW alignment.

Problem

• cross-tokenizer knowledge distillation
• sequence alignment
• vocabulary alignment
• large language model compression
• token-wise distillation

Innovation

• Dual-Space Weighting
• Time-Warped Alignment
• Cross-Tokenizer Knowledge Distillation
• Soft-DTW
• Entropy-Based Token Weighting