Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it

2026-08-24

AI Summary

本文探讨了知识蒸馏在中间层应用时可能导致过拟合的问题，并提出了一种基于同构块的学生网络设计方法，通过适当的指导策略来避免这一问题。

Abstract

The growing size of Convolutional Neural Networks has led to increasingly large and costly models. Knowledge Distillation (KD) addresses this by transferring knowledge from a large network (teacher) to a small one (student), also reducing the training data required. KD is traditionally applied only at the network's final output. However, its behaviour when applied at intermediate network layers has received little attention. This raises the question of whether intermediate block-wise KD, which provides supervision throughout the network, could offer an advantage under specific conditions, such as few instances per class, which is common in fine-grained datasets. This work proposes a student design based on simple, homogeneous blocks mirroring those of the teacher, distilling knowledge between corresponding blocks. Across eleven datasets, we show that on classic datasets, distilling only the last block is sufficient -- and often best--, whereas fine-grained, data-scarce settings benefit substantially from intermediate supervision, with even a single additional distillation point narrowing the gap considerably. We further study how this supervision should be guided, exploring configurations of varying granularity and informed by an explainability analysis based on attention maps, Centered Kernel Alignment, and Grad-CAM, alongside the impact of teacher and student fine-tuning strategies. This work shows that intermediate block-wise distillation, guided appropriately, is key to building compact data-efficient models without sacrificing accuracy.

Problem

• Knowledge Distillation
• overfitting
• intermediate layers
• fine-grained datasets
• data scarcity

Innovation

• Intermediate Block-wise Distillation
• Fine-Grained Datasets
• Attention Maps
• Centered Kernel Alignment
• Grad-CAM