🤖 AI Summary
To address distribution misalignment in generative tasks, overfitting in low-probability regions, and severe exposure bias in knowledge distillation for open-source white-box large language models (LLMs), this work introduces reverse KL divergence—first applied to LLM distillation—to formulate a generation-aware optimization objective and develop an efficient solver. We further propose an instruction-tuning distillation framework coupled with multi-scale model adaptation, enabling effective distillation across model sizes from 120M to 13B parameters. Experiments demonstrate substantial improvements in output calibration, long-text coherence, and instruction-following capability, consistently outperforming state-of-the-art baselines across response accuracy, overall quality, and other key metrics. All code, datasets, and distilled models are publicly released.
📝 Abstract
Knowledge Distillation (KD) is a promising technique for reducing the high computational demand of large language models (LLMs). However, previous KD methods are primarily applied to white-box classification models or training small models to imitate black-box model APIs like ChatGPT. How to effectively distill the knowledge of white-box LLMs into small models is still under-explored, which becomes more important with the prosperity of open-source LLMs. In this work, we propose a KD approach that distills LLMs into smaller language models. We first replace the forward Kullback-Leibler divergence (KLD) objective in the standard KD approaches with reverse KLD, which is more suitable for KD on generative language models, to prevent the student model from overestimating the low-probability regions of the teacher distribution. Then, we derive an effective optimization approach to learn this objective. The student models are named MiniLLM. Extensive experiments in the instruction-following setting show that MiniLLM generates more precise responses with higher overall quality, lower exposure bias, better calibration, and higher long-text generation performance than the baselines. Our method is scalable for different model families with 120M to 13B parameters. Our code, data, and model checkpoints can be found in https://github.com/microsoft/LMOps/tree/main/minillm.
