Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations

2026-07-13

AI Summary

This work addresses the limitations of conventional single-teacher knowledge distillation, which suffers from restricted performance due to its reliance on a single supervisory signal, while multi-teacher or augmentation-based approaches often incur additional computational overhead, multi-stage training, or parameter growth. To overcome these issues, the authors propose Shift-Augmented Knowledge Distillation (SAKD), a single-stage, end-to-end framework that generates diverse and adaptive supervisory views by applying parameter-free cyclic shift perturbations to the teacher’s output, conditioned on the student model’s dynamic features. SAKD requires no extra parameters or pretraining and significantly outperforms random perturbation baselines. It achieves accuracy on par with two-stage methods on CIFAR-100 and ImageNet while substantially reducing model complexity.

Abstract

Knowledge distillation (KD) typically relies on the fixed perspective of a single teacher, limiting the diversity of supervisory signals. While multi-teacher distillation addresses this by aggregating knowledge from multiple models, it incurs prohibitive computational and storage costs. To balance efficiency and diversity, recent research has focused on generating virtual views from a single teacher. However, existing methods face a trade-off: random perturbation approaches offer efficiency but lack controlled diversity, while structured augmentation methods require multi-stage training and incur linear parameter growth. We observe that this trade-off stems from a common design choice: using the teacher's strong but static features to generate views. Instead, we propose Shift-Augmented Knowledge Distillation (SAKD), a simple yet effective framework that leverages the student's evolving features as a dynamic condition for perturbation generation. This shift in perspective enables single-stage training while producing adaptive, diverse views through a parameter-free cyclic shift. Extensive experiments on CIFAR-100 and ImageNet demonstrate that SAKD consistently outperforms random perturbation methods and achieves accuracy on par with two-stage approaches, while using significantly fewer parameters and eliminating pre-training requirements.

Problem

• knowledge distillation
• view augmentation
• single-teacher
• diversity-efficiency trade-off
• supervisory signal diversity

Innovation

• knowledge distillation
• view augmentation
• student-guided perturbation
• single-stage training
• parameter-free