Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails

2026-06-22

AI Summary

This work investigates the role of post-training knowledge distillation in building efficient small language models under data-scarce or resource-constrained settings. Through systematic analysis across varying data scales and teacher model strengths, the study demonstrates that knowledge distillation significantly outperforms supervised fine-tuning in low-data regimes, though this advantage diminishes as data volume increases. To address this limitation, the authors propose a two-stage distillation strategy that combines synthetic data with human-annotated examples, consistently enhancing student model performance on domain-specific tasks. Experiments on the Tulu 3 dataset further reveal that stronger instruction-tuned teacher models can restore distillation’s superiority even at higher data scales, offering a practical and effective model compression approach for resource-limited environments.

Abstract

Large language models (LLMs) achieve strong performance across many tasks, but their high computational cost limits deployment in resource-constrained environments. Knowledge Distillation (KD) offers a practical solution by transferring knowledge from a teacher model of a larger size to a smaller student model. While prior work has mainly examined task-specific or small-scale settings, the post-training stage for building general instruction-following models has received limited attention. In this paper, we conduct a systematic study of KD in post-training using the large-scale Tulu 3 dataset. We find that KD outperforms supervised fine-tuning (SFT) in low-data regimes, but its advantage diminishes as more training data is added. Distilling from a stronger instruction-tuned teacher restores substantial gains even with abundant data, indicating that KD remains effective when the teacher provides knowledge that the student cannot easily acquire from the training data alone. We further study domain-specific, low-resource scenarios and propose a two-stage KD strategy that leverages synthetic teacher-labeled data followed by refinement on human annotations. This method consistently improves student performance, providing practical guidance for building compact models in data-scarce environments.

Problem

• Knowledge Distillation
• Post-Training
• Large Language Models
• Data Scarcity
• Instruction Tuning

Innovation

• Knowledge Distillation
• Post-Training
• Instruction Tuning
• Low-Resource Learning
• Two-Stage Distillation