Generative Distribution Distillation

2025-07-19

AI Summary

Addressing two key challenges in knowledge distillation (KD)—high-dimensional optimization difficulty and lack of label-semantic supervision—this paper pioneers a conditional generative formulation of KD, proposing the Generative Distribution Distillation (GDD) framework. Methodologically, GDD introduces three core innovations: (1) a split-tokenization strategy enabling stable unsupervised distillation; (2) distribution contraction, a theoretically grounded technique proven equivalent to multi-task gradient proxy, which implicitly injects label supervision without explicit classification loss; and (3) support for efficient multi-step sampling during training. Under the ImageNet unsupervised setting, GDD surpasses the KL-divergence baseline by 16.29% in top-1 accuracy. With supervised training on ImageNet, ResNet-50 distilled via GDD achieves 82.28% top-1 accuracy—the highest reported among comparable KD methods.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Generation
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

In this paper, we formulate the knowledge distillation (KD) as a conditional generative problem and propose the extit{Generative Distribution Distillation (GenDD)} framework. A naive extit{GenDD} baseline encounters two major challenges: the curse of high-dimensional optimization and the lack of semantic supervision from labels. To address these issues, we introduce a extit{Split Tokenization} strategy, achieving stable and effective unsupervised KD. Additionally, we develop the extit{Distribution Contraction} technique to integrate label supervision into the reconstruction objective. Our theoretical proof demonstrates that extit{GenDD} with extit{Distribution Contraction} serves as a gradient-level surrogate for multi-task learning, realizing efficient supervised training without explicit classification loss on multi-step sampling image representations. To evaluate the effectiveness of our method, we conduct experiments on balanced, imbalanced, and unlabeled data. Experimental results show that extit{GenDD} performs competitively in the unsupervised setting, significantly surpassing KL baseline by extbf{16.29%} on ImageNet validation set. With label supervision, our ResNet-50 achieves extbf{82.28%} top-1 accuracy on ImageNet in 600 epochs training, establishing a new state-of-the-art.

Problem

• Addresses high-dimensional optimization in knowledge distillation
• Lacks semantic supervision from labels in distillation
• Improves unsupervised and supervised distillation performance

Innovation

• Formulates KD as conditional generative problem
• Introduces Split Tokenization for stable KD
• Develops Distribution Contraction for label supervision