Strong Teacher Not Needed? On Distillation in LLM Pretraining

2026-05-22

AI Summary

This study challenges the conventional assumption that knowledge distillation in large language model pretraining necessarily requires a strong teacher model. It systematically investigates distillation efficacy across varying teacher-student capacity pairings—including strong-to-weak, peer-level, and weak-to-strong configurations—by jointly optimizing language modeling and distillation losses. Evaluated across diverse Transformer architectures and out-of-distribution downstream tasks, the findings reveal that even weak teachers can significantly enhance student performance when the loss weighting is appropriately calibrated. Moreover, stronger teachers do not consistently yield better distillation outcomes; excessive teacher capacity may lead to performance saturation or even degradation. The work thus uncovers the underappreciated potential of weak-teacher distillation and highlights its beneficial impact on out-of-domain generalization.

Abstract

Knowledge distillation generally assumes a strong-to-weak relationship where stronger teachers yield better students. In this work, we examine this assumption about distillation in large language model pretraining. By varying architecture sizes and training token budgets, we create strong-to-weak, same-level, and weak-to-strong teacher-student relationships, and study distillation's effectiveness under each. We find that the teacher need not be strong: with proper mixing of the language modeling and knowledge distillation losses, even small and undertrained teachers improve larger students. At the same time, a stronger teacher is not always better: pushing the teacher further, through more parameters or more training tokens, can saturate or even reverse the distillation gains. We further observe that distillation improves generalization (out-of-distribution and downstream performance) more readily than in-domain fitting. Together, these results challenge the common belief that distillation pretraining always requires a strong teacher.

Problem

• knowledge distillation
• large language models
• pretraining
• teacher-student relationship
• model generalization

Innovation

• knowledge distillation
• large language models
• teacher-student paradigm
• pretraining
• generalization