CAE-DFKD: Bridging the Transferability Gap in Data-Free Knowledge Distillation

2025-04-30

AI Summary

Existing data-free knowledge distillation (DFKD) methods focus on image-level recognition performance while neglecting the transferability of embedding representations. This work addresses this limitation by proposing the first DFKD framework explicitly designed for embedding-level transferability—introducing a category-aware synthetic data generation and knowledge transfer mechanism directly in the embedding space, thereby departing from conventional image-level distillation paradigms. Our approach comprises five key components: (i) category-aware embedding constraints, (ii) decoupled generator training, (iii) teacher-guided feature distribution matching, (iv) contrastive learning enhancement, and (v) a lightweight generator design. Evaluated on standard image classification benchmarks, the method achieves state-of-the-art performance; it further improves downstream classification and detection tasks by an average of 12.6% and accelerates synthetic data generation by 3.2×. This work establishes a novel paradigm for modeling transferable representations in DFKD.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Data-Free Knowledge Distillation (DFKD) enables the knowledge transfer from the given pre-trained teacher network to the target student model without access to the real training data. Existing DFKD methods focus primarily on improving image recognition performance on associated datasets, often neglecting the crucial aspect of the transferability of learned representations. In this paper, we propose Category-Aware Embedding Data-Free Knowledge Distillation (CAE-DFKD), which addresses at the embedding level the limitations of previous rely on image-level methods to improve model generalization but fail when directly applied to DFKD. The superiority and flexibility of CAE-DFKD are extensively evaluated, including: extit{ extbf{i.)}} Significant efficiency advantages resulting from altering the generator training paradigm; extit{ extbf{ii.)}} Competitive performance with existing DFKD state-of-the-art methods on image recognition tasks; extit{ extbf{iii.)}} Remarkable transferability of data-free learned representations demonstrated in downstream tasks.

Problem

• Addresses transferability gap in data-free knowledge distillation
• Improves model generalization at embedding level
• Enhances efficiency and performance in image recognition

Innovation

• Category-Aware Embedding improves DFKD transferability
• Generator training paradigm enhances efficiency
• Competitive performance in image recognition tasks