Soft Label Pruning and Quantization for Large-Scale Dataset Distillation

2026-04-20

AI Summary

This work addresses the prohibitive storage overhead of auxiliary soft labels in large-scale dataset distillation, which severely limits compression efficiency. To mitigate reliance on soft labels, the authors propose an efficient distillation framework that enhances the diversity of synthetic images and supervision signals through intra-class batching and batch normalization-based supervision. Additionally, they introduce a dynamic knowledge reuse strategy comprising label pruning and a calibration-aware label quantization scheme to align teacher–student predictions effectively. Evaluated on ImageNet-1K and ImageNet-21K, the method reduces soft label storage costs by up to 78× and 500×, respectively, while simultaneously improving model accuracy by as much as 7.2% and 2.8%, thereby achieving an exceptional balance between compression ratio and model performance.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Computer Vision: Diffusion Models for Vision
• Data Mining & Knowledge Management: Data Compression

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Large-scale dataset distillation requires storing auxiliary soft labels that can be 30-40x larger on ImageNet-1K and 200x larger on ImageNet-21K than the condensed images, undermining the goal of dataset compression. We identify two fundamental issues necessitating such extensive labels: (1) insufficient image diversity, where high within-class similarity in synthetic images requires extensive augmentation, and (2) insufficient supervision diversity, where limited variety in supervisory signals during training leads to performance degradation at high compression rates. To address these challenges, we propose Label Pruning and Quantization for Large-scale Distillation (LPQLD). We enhance image diversity via class-wise batching and batch-normalization supervision during synthesis. For supervision diversity, we introduce Label Pruning with Dynamic Knowledge Reuse to improve label-per-augmentation diversity, and Label Quantization with Calibrated Student-Teacher Alignment to improve augmentation-per-image diversity. Our approach reduces soft label storage by 78x on ImageNet-1K and 500x on ImageNet-21K while improving accuracy by up to 7.2% and 2.8%, respectively. Extensive experiments validate the superiority of LPQLD across different network architectures and dataset distillation methods. Code is available at https://github.com/he-y/soft-label-pruning-quantization-for-dataset-distillation.

Problem

• dataset distillation
• soft labels
• label storage
• image diversity
• supervision diversity

Innovation

• Label Pruning
• Label Quantization
• Dataset Distillation
• Soft Labels
• Supervision Diversity