🤖 AI Summary
Existing knowledge distillation (KD) methods rely heavily on Kullback–Leibler (KL) divergence, which suffers from insufficient or biased knowledge transfer under high- or low-entropy teacher distributions; moreover, standard data augmentation can inadvertently degrade KD performance. To address these issues, we propose a robust KD framework comprising three key innovations: (i) replacing KL divergence with correlation-based distance to mitigate entropy sensitivity; (ii) integrating structured network pruning to enhance student model robustness; and (iii) identifying and mitigating the adverse interference of data augmentation in KD via a multi-stage teacher–student co-optimization strategy. Extensive experiments on CIFAR-100, FGVC-Aircraft, TinyImageNet, and ImageNet demonstrate state-of-the-art performance: student models achieve average accuracy gains of 1.2–2.7% over prior methods and exhibit significantly improved resilience to input noise and adversarial perturbations.
📝 Abstract
The improvement in the performance of efficient and lightweight models (i.e., the student model) is achieved through knowledge distillation (KD), which involves transferring knowledge from more complex models (i.e., the teacher model). However, most existing KD techniques rely on Kullback-Leibler (KL) divergence, which has certain limitations. First, if the teacher distribution has high entropy, the KL divergence's mode-averaging nature hinders the transfer of sufficient target information. Second, when the teacher distribution has low entropy, the KL divergence tends to excessively focus on specific modes, which fails to convey an abundant amount of valuable knowledge to the student. Consequently, when dealing with datasets that contain numerous confounding or challenging samples, student models may struggle to acquire sufficient knowledge, resulting in subpar performance. Furthermore, in previous KD approaches, we observed that data augmentation, a technique aimed at enhancing a model's generalization, can have an adverse impact. Therefore, we propose a Robustness-Reinforced Knowledge Distillation (R2KD) that leverages correlation distance and network pruning. This approach enables KD to effectively incorporate data augmentation for performance improvement. Extensive experiments on various datasets, including CIFAR-100, FGVR, TinyImagenet, and ImageNet, demonstrate our method's superiority over current state-of-the-art methods.