SFKD: Spatial--Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral Interaction

2026-07-02

AI Summary

This work addresses the limitations of existing knowledge distillation methods, which are predominantly confined to homogeneous architectures and struggle to handle the distributional misalignment between heterogeneous teacher and student models caused by differing inductive biases, often leading to the loss of critical structural semantics and local details. To overcome this, we propose SFKD, a novel framework that, for the first time, jointly models spatial and frequency-domain information in heterogeneous distillation. SFKD explicitly decouples and preserves fine-grained local details via multi-level discrete wavelet transforms, captures global energy distributions through a Fourier-domain Gaussian filtering loss, and employs a dual-stream, two-stage refinement module to enable joint spatial-frequency optimization. Extensive experiments demonstrate that SFKD significantly outperforms state-of-the-art methods across multiple benchmark datasets, exhibiting both strong effectiveness and generalization capability.

Abstract

Most existing knowledge distillation methods focus on homogeneous models (e.g., CNN-to-CNN), thereby overlooking the flexibility and potential of knowledge transfer across heterogeneous models. Due to intrinsic inductive bias discrepancies between heterogeneous models that cause spatial distribution inconsistencies, prior heterogeneous distillation methods often weaken or discard spatial information in heterogeneous representations. However, the spatial information in representations often encodes transferable global structural semantics as well as architecture-specific local details, and therefore should not be directly ignored. To better leverage the spatial information encoded in heterogeneous representations, we propose a Spatial-Frequency Joint-Aware Heterogeneous Knowledge Distillation framework (SFKD). By leveraging the complementary properties of wavelet transform spatial locality and Fourier representations in characterizing global energy distributions, we first apply multi-level discrete wavelet transform to explicitly decouple spatial information. The resulting wavelet sub-bands are further refined by a dual-stream dual-stage refinement module, and finally combined with a Gaussian-filtered frequency loss to selectively capture informative global information. Extensive experiments on multiple benchmark datasets under both homogeneous and heterogeneous models demonstrate the superiority of our method.

Problem

• heterogeneous knowledge distillation
• spatial information
• inductive bias discrepancy
• representation transfer
• model heterogeneity

Innovation

• heterogeneous knowledge distillation
• spatial-frequency joint awareness
• multi-level wavelet transform
• dual-stream refinement
• frequency-aware loss