Knowledge Distillation Must Account for What It Loses

2026-04-27

AI Summary

While knowledge distillation often preserves task performance in student models, it frequently overlooks degradation in critical capabilities such as uncertainty calibration, boundary behavior, and safety. This work reframes distillation as a lossy projection of teacher behavior and introduces the “Distillation Loss Statement” framework, which integrates context-specific capability preservation objectives to establish a measurable and accountable evaluation paradigm. Through behavioral fidelity analysis, a capability taxonomy, and multidimensional assessment, the study systematically identifies and quantifies non-task-related capability losses. The resulting reproducible taxonomy of distillation-induced losses advances the field beyond mere performance retention toward practical standards that jointly prioritize reliability and responsibility in distilled models.

Abstract

This position paper argues that knowledge distillation must account for what it loses: student models should be judged not only by retained task scores, but by whether they preserve the teacher capabilities that make those scores reliable. This matters because distillation is increasingly used to turn large, often frontier models into deployable systems, yet headline metrics can hide losses in uncertainty, boundary behavior, process reliability, on-policy stability, grounding, privacy, safety, and diversity. We identify the retention assumption behind current evaluation and reframe distillation as a lossy projection of teacher behavior rather than a faithful copy. We then synthesize existing evidence into a taxonomy of off-metric distillation losses, showing that these losses are concrete, recurring, and measurable. To make the position actionable, we propose scenario-specific preservation targets and a Distillation Loss Statement that reports what was preserved, what was lost, and why the remaining losses are acceptable. The goal is not lossless distillation, but accountable distillation.

Problem

• knowledge distillation
• capability preservation
• distillation loss
• model reliability
• evaluation metrics

Innovation

• knowledge distillation
• distillation loss
• accountable AI
• behavioral preservation
• lossy projection