Knowledge Distillation with Refined Logits

2024-08-14

arXiv.org

AI Summary

Existing knowledge distillation methods suffer from conflicting distillation targets when teacher predictions are erroneous, and hard-label correction disrupts inter-class semantic correlations. To address this, we propose Refined Logit Distillation (RLD), the first framework introducing a dynamic logit refinement mechanism: a label-guided, differentiable logit recalibration that jointly applies confidence weighting and temperature-adaptive scaling—enabling erroneous predictions to be corrected while fully preserving inter-class semantic structure. RLD employs a hybrid loss combining cross-entropy and KL divergence, mitigating semantic distortion inherent in conventional logit-based distillation. Extensive experiments on CIFAR-100 and ImageNet demonstrate consistent improvements, with student models achieving average Top-1 accuracy gains of 1.2%–2.3% over strong baselines including KD, RKD, and VID. The implementation is publicly available.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Description Logics
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Recent research on knowledge distillation has increasingly focused on logit distillation because of its simplicity, effectiveness, and versatility in model compression. In this paper, we introduce Refined Logit Distillation (RLD) to address the limitations of current logit distillation methods. Our approach is motivated by the observation that even high-performing teacher models can make incorrect predictions, creating a conflict between the standard distillation loss and the cross-entropy loss. This conflict can undermine the consistency of the student model's learning objectives. Previous attempts to use labels to empirically correct teacher predictions may undermine the class correlation. In contrast, our RLD employs labeling information to dynamically refine teacher logits. In this way, our method can effectively eliminate misleading information from the teacher while preserving crucial class correlations, thus enhancing the value and efficiency of distilled knowledge. Experimental results on CIFAR-100 and ImageNet demonstrate its superiority over existing methods. The code is provided at ext{https://github.com/zju-SWJ/RLD}.

Problem

• Address limitations of current logit distillation methods
• Eliminate misleading teacher predictions while preserving class correlations
• Enhance value and efficiency of distilled knowledge

Innovation

• Dynamic refinement of teacher logits
• Preserves crucial class correlations
• Eliminates misleading teacher information