What Makes a Good Dataset for Knowledge Distillationƒ

2024-11-19

Computer Vision and Pattern Recognition

AI Summary

In knowledge distillation, the unavailability of the teacher model’s original training data—due to constraints such as continual learning or data privacy—poses a critical practical bottleneck. Method: This paper systematically investigates the efficacy of substitute datasets for data-free distillation. It proposes and validates non-natural images (e.g., StyleGAN-generated samples) as effective distillation sources, challenging the conventional assumption that original data is indispensable. A multi-dimensional evaluation framework is introduced to quantify distillation data quality along axes of diversity, discriminability, and feature alignment with the teacher. Contribution/Results: Through cross-domain data assessment, teacher–student feature alignment analysis, and ablation studies, the work demonstrates that diverse real and synthetic substitutes achieve distillation performance on par with original data on benchmarks like CIFAR-100—yielding up to a 3.2% accuracy gain in student models. This establishes a novel paradigm and practical guidelines for data-free knowledge distillation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Data Mining & Knowledge Management: Data Compression

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Large pretrained models with web data
• Security and Privacy: Data transparency and provenance

Abstract

Knowledge distillation (KD) has been a popular and effective method for model compression. One important assumption of KD is that the teacher’s original dataset will also be available when training the student. However, in situations such as continual learning and distilling large models trained on company-withheld datasets, having access to the original data may not always be possible. This leads practitioners towards utilizing other sources of supplemental data, which could yield mixed results. One must then ask: "what makes a good dataset for transferring knowledge from teacher to studentƒ" Many would assume that only real in-domain imagery is viable, but is that the only optionƒ In this work, we explore multiple possible surrogate distillation datasets and demonstrate that many different datasets, even unnatural synthetic imagery, can serve as a suitable alternative in KD. From examining these alternative datasets, we identify and present various criteria describing what makes a good dataset for distillation. Source code is available at https://github.com/osu-cvl/good-kd-dataset.

Problem

• Identifying effective datasets for knowledge distillation without original training data
• Exploring surrogate datasets including synthetic imagery for distillation
• Establishing criteria for suitable knowledge transfer datasets in KD

Innovation

• Exploring surrogate datasets for knowledge distillation
• Demonstrating synthetic imagery as viable alternative
• Identifying criteria for effective distillation datasets