Knowledge Distillation with Adapted Weight

2025-01-06

AI Summary

To address the low efficiency, poor stability, and weak interpretability of knowledge distillation (KD) for deploying large language models on resource-constrained devices (e.g., smartphones), this paper proposes KD-AIF, an adaptive data-weighting KD framework grounded in influence functions. It is the first work to incorporate influence functions—originally from robust statistics—into KD, enabling dynamic, principle-driven sample weighting based on the SAFE criteria (Sustainability, Accuracy, Fairness, Explainability). KD-AIF integrates a teacher-student architecture, adaptive weight updating, and semi-supervised collaborative optimization. Evaluated on CIFAR-100, CIFAR-10-4k, SVHN-1k, and GLUE benchmarks, KD-AIF consistently outperforms state-of-the-art methods, achieving superior generalization and model interpretability while maintaining high accuracy.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Natural Language Processing: Safety and Robustness
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Although large models have shown a strong capacity to solve large-scale problems in many areas including natural language and computer vision, their voluminous parameters are hard to deploy in a real-time system due to computational and energy constraints. Addressing this, knowledge distillation through Teacher-Student architecture offers a sustainable pathway to compress the knowledge of large models into more manageable sizes without significantly compromising performance. To enhance the robustness and interpretability of this framework, it is critical to understand how individual training data impact model performance, which is an area that remains underexplored. We propose the extbf{Knowledge Distillation with Adaptive Influence Weight (KD-AIF)} framework which leverages influence functions from robust statistics to assign weights to training data, grounded in the four key SAFE principles: Sustainability, Accuracy, Fairness, and Explainability. This novel approach not only optimizes distillation but also increases transparency by revealing the significance of different data. The exploration of various update mechanisms within the KD-AIF framework further elucidates its potential to significantly improve learning efficiency and generalization in student models, marking a step toward more explainable and deployable Large Models. KD-AIF is effective in knowledge distillation while also showing exceptional performance in semi-supervised learning with outperforms existing baselines and methods in multiple benchmarks (CIFAR-100, CIFAR-10-4k, SVHN-1k, and GLUE).

Problem

• Efficient Learning
• Model Performance
• Resource-constrained Environment

Innovation

• Knowledge Distillation
• Adaptive Importance Weighting
• Semi-supervised Learning