Distillation Traps and Guards: A Calibration Knob for LLM Distillability

2026-04-20

AI Summary

This work addresses the challenges in large language model knowledge distillation—namely, tail noise, off-policy instability, teacher-student capability gaps, and risks of model leakage—by proposing a post-hoc calibration method that formalizes “distillability” as a controllable safety knob. Through reinforcement fine-tuning (RFT), the approach dynamically modulates teacher outputs by integrating task utility, KL anchoring, and cross-tokenizer calibration rewards. This enables simultaneous preservation of teacher performance and significant enhancement—or intentional suppression—of student learning. Empirical results demonstrate that students trained with calibrated teachers consistently outperform those from supervised fine-tuning (SFT) and standard knowledge distillation baselines across mathematical reasoning, question answering, and instruction-following tasks. Conversely, when distillability is disabled, student performance collapses, confirming the method’s dual efficacy in boosting distillation efficiency and safeguarding model intellectual property.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Knowledge distillation (KD) transfers capabilities from large language models (LLMs) to smaller students, yet it can fail unpredictably and also underpins model leakage risks. Our analysis revealed several distillation traps: tail noise, off-policy instability, and, most fundamentally, the teacher-student gap, that distort training signals. These traps manifest as overconfident hallucinations, self-correction collapse, and local decoding degradation, causing distillation to fail. Motivated by these findings, we propose a post-hoc calibration method that, to the best of our knowledge, for the first time enables control over a teacher's distillability via reinforcement fine-tuning (RFT). Our objective combines task utility, KL anchor, and across-tokenizer calibration reward. This makes distillability a practical safety lever for foundation models, connecting robust teacher-student transfer with deployment-aware model protection. Experiments across math, knowledge QA, and instruction-following tasks show that students distilled from distillable calibrated teachers outperform SFT and KD baselines, while undistillable calibrated teachers retain their task performance but cause distilled students to collapse, offering a practical knob for both better KD and model IP protection.

Problem

• knowledge distillation
• distillation traps
• teacher-student gap
• model leakage
• LLM distillability

Innovation

• distillability
• knowledge distillation
• reinforcement fine-tuning
• model calibration
• LLM security