ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization

2026-08-30

AI Summary

本文提出ACTD方法，通过词汇和序列对齐及残差正则化解决跨分词器知识蒸馏中的异构性和噪声问题，提高轻量模型性能。

Abstract

Knowledge distillation effectively transfers reasoning capabilities from large language models to lightweight student models. To enable knowledge transfer across disparate model families, researchers increasingly explore cross-tokenizer distillation. However, cross-tokenizer distillation remains challenging due to vocabulary and sequence misalignment, while approximate vocabulary alignment can introduce additional noise into distillation. To address these challenges, we propose Anchor-Based Cross-Tokenizer Distillation with Residual Regularization (ACTD). ACTD bridges structural heterogeneity through vocabulary and sequence alignment, while mitigating alignment noise via a novel anchor loss with residual regularization. We further extend this framework to a multi-teacher setting. Evaluated across five reasoning benchmarks with three distinct teacher models, ACTD achieves state-of-the-art performance. Moreover, its multi-teacher extension outperforms the strongest single-teacher and multi-teacher baselines, further demonstrating the robustness of our method.

Problem

• cross-tokenizer distillation
• vocabulary misalignment
• sequence misalignment
• alignment noise

Innovation

• Anchor-Based Cross-Tokenizer Distillation
• Residual Regularization
• Multi-teacher Setting