machine learning methods

Existing knowledge distillation methods struggle to model the structured relationships among internal representations of teacher models, while mainstream contrastive learning objectives (e.g., InfoNCE) impose overly stringent instance discrimination constraints, disrupting relative semantic similarities among semantically proximal samples. To address these limitations, we propose Relational Representation Distillation (RRD). Its core innovations are: (1) a dual-temperature Softmax mechanism—employing a high temperature to emphasize dominant relational patterns and a low temperature to preserve secondary semantic similarities; and (2) a theoretically unified loss that bridges InfoNCE and KL divergence, enabling relative distribution alignment. Evaluated on multi-task transfer learning benchmarks, RRD significantly improves teacher–student representation alignment. Notably, on several downstream tasks, student models trained with RRD even surpass their teachers in performance—demonstrating both the effectiveness of structured relational modeling and its strong generalization capability.

To address weak semantic preservation and degraded reasoning performance in large language model (LLM) knowledge distillation—caused by teacher-student representation mismatch—this paper proposes a novel distillation framework integrating feature alignment with hierarchical representation transfer. Methodologically, it innovatively couples fine-grained hidden-layer feature-space alignment via contrastive learning, gradient-aware dynamic scheduling of representation transfer weights, and a modular decoupled distillation mechanism—thereby overcoming the locality limitations of conventional logit- or attention-based distillation and enabling cross-depth semantic consistency modeling. Evaluated on LLaMA-2 → TinyLLaMA distillation, the student model achieves 92.3% of the teacher’s original accuracy despite an 78% reduction in parameter count, while attaining a 3.1× speedup in inference latency. These results significantly outperform existing state-of-the-art methods.

This study addresses the underutilization of knowledge distillation (KD) for pretrained models in distributed and federated learning settings, systematically evaluating multiple KD variants under heterogeneous data distributions. We propose the first lightweight, practical KD framework tailored for federated learning, incorporating multi-strategy data partitioning and hyperparameter sensitivity analysis to uncover adaptation patterns for critical hyperparameters—such as temperature scaling and loss weighting. Innovatively, we integrate tuned KD, deep mutual learning, and data-partitioned KD, jointly optimized via grid search. Experiments demonstrate that our approach significantly reduces communication rounds and accelerates model convergence in federated learning. Moreover, it delivers reusable, scenario-specific optimal KD configurations across diverse data partitioning schemes, boosting student model accuracy by 3.2–5.7% on average.

Conventional knowledge distillation tightly couples teacher and student architectures, resulting in poor cross-architecture generalization and prohibitive retraining costs for each new student. Method: We propose the Generalized Teacher Network (GTN), the first architecture-agnostic teacher framework that models the student pool as a weight-sharing supernet and employs a capacity-aware conditional mechanism to dynamically adapt the teacher to diverse student architectures. GTN jointly trains the teacher and students in a single, distillation-aware optimization pass. Contribution/Results: GTN eliminates the need for per-student teacher training; its overhead is amortized across the student pool. Evaluated on multi-architecture student pools, GTN consistently improves accuracy by 1.2–2.8% over baseline distillation methods, significantly enhancing deployment flexibility and computational efficiency.

Existing knowledge distillation (KD) methods rely heavily on Kullback–Leibler (KL) divergence, which suffers from insufficient or biased knowledge transfer under high- or low-entropy teacher distributions; moreover, standard data augmentation can inadvertently degrade KD performance. To address these issues, we propose a robust KD framework comprising three key innovations: (i) replacing KL divergence with correlation-based distance to mitigate entropy sensitivity; (ii) integrating structured network pruning to enhance student model robustness; and (iii) identifying and mitigating the adverse interference of data augmentation in KD via a multi-stage teacher–student co-optimization strategy. Extensive experiments on CIFAR-100, FGVC-Aircraft, TinyImageNet, and ImageNet demonstrate state-of-the-art performance: student models achieve average accuracy gains of 1.2–2.7% over prior methods and exhibit significantly improved resilience to input noise and adversarial perturbations.

Although knowledge distillation is widely employed to enhance model generalization, its theoretical underpinnings remain poorly understood. This work models the teacher–student training dynamics as a coupled stochastic process and introduces a novel “distillation divergence” to quantify the discrepancy between teacher and student. Building upon this, we develop an information-theoretic framework for generalization analysis and derive upper and lower bounds on the student’s generalization error that explicitly depend on the distillation divergence. Notably, we show that the local flatness of the teacher model strictly tightens the upper bound. In the Gaussian linear setting, we further provide an interpretable decomposition of the error into bias, variance, and a rank bottleneck, offering both theoretical insights and practical principles for designing effective distillation algorithms.

This work addresses the limitations of existing knowledge distillation methods when integrating multiple heterogeneous strategies, which often suffer from implementation complexity, rigid combinations, and catastrophic forgetting. To overcome these challenges, the authors propose a Sequential Multi-Stage Knowledge Distillation (SMSKD) framework that applies distinct distillation techniques in successive stages. Each stage leverages a frozen reference model from the previous stage to anchor learned knowledge, while a sample-level adaptive loss weighting mechanism—based on the teacher’s true class probability (TCP)—dynamically balances knowledge retention and integration. The framework flexibly accommodates arbitrary distillation strategies and numbers of stages, consistently yielding significant accuracy improvements for student models across diverse teacher–student architectures, outperforming current baselines with negligible computational overhead.

This study investigates dataset size—a long-overlooked critical variable—in knowledge distillation (KD). Addressing the limited explanatory power of existing theories (e.g., label smoothing and dark knowledge hypotheses) across varying data regimes, we design a systematic experimental framework spanning multiple tasks, architectures, and datasets, with strict control over sample count and model scale, and analyze dynamic distillation loss behavior. Results reveal KD’s pronounced data efficiency: performance gains are substantially larger under data-scarce conditions. The label smoothing hypothesis is empirically refuted, whereas the dark knowledge hypothesis receives stronger empirical support. Crucially, this work establishes dataset size as a fundamental determinant of KD effectiveness and theoretical interpretability—challenging prevailing assumptions in distillation literature. It provides the first rigorous evidence that data scale governs both KD’s empirical success and its underlying mechanism, thereby enabling principled theory refinement and advancing KD deployment in low-resource settings.

This work proposes a systematic method to convert non-neural machine learning pipelines—such as those based on random forests—into neural networks, enabling unified inference and joint optimization. Leveraging knowledge distillation, the approach treats the traditional model as a “teacher” that guides the training of a neural “student” network. The framework further integrates neural architecture search with a random forest–inspired hyperparameter selection strategy to optimize the student model. Notably, this is the first effort to employ an entire non-neural machine learning pipeline as the teacher in knowledge distillation, thereby extending the scope of this technique. Experimental evaluation across 100 OpenML tasks demonstrates that the student networks consistently replicate the performance of their teacher models, confirming the feasibility and effectiveness of the proposed conversion framework.

Existing feature-based knowledge distillation (KD) methods still rely on logit-level losses (e.g., cross-entropy), hindering effective transfer of intermediate-layer feature knowledge. Method: We propose the first purely feature-driven KD framework that completely eliminates logit supervision. Instead, it trains student backbone networks via intermediate-feature alignment and geometric analysis of latent-space representations. We introduce a novel metric to quantitatively assess feature knowledge quality, enabling adaptive selection of optimal teacher layers; additionally, we design a distribution-aware alignment loss grounded in feature geometry to enhance representation consistency. Contribution/Results: Our method achieves significant improvements over state-of-the-art approaches on three image classification benchmarks, with up to 15% absolute Top-1 accuracy gain. It is the first work to empirically validate both the feasibility and superiority of high-fidelity feature knowledge transfer without any logit-level supervision.