From Fewer Samples to Fewer Bits: Reframing Dataset Distillation as Joint Optimization of Precision and Compactness

2026-03-02

AI Summary

This work addresses the limitation of existing dataset distillation methods, which primarily focus on reducing the number of synthetic samples while neglecting the impact of numerical precision on information efficiency. To overcome this, we propose QuADD (Quantization-Aware Dataset Distillation), a novel framework that jointly optimizes both the quantity and precision of synthetic samples under a fixed bit budget—unifying these two aspects into a single optimization objective for the first time. QuADD integrates a differentiable quantization module, adaptive non-uniform quantization, and a rate-distortion theory-guided bit allocation strategy to enable end-to-end joint optimization. Experiments demonstrate that QuADD significantly outperforms current distillation and post-quantization approaches in terms of accuracy per bit on both image classification and 3GPP beam management tasks, establishing a new paradigm for efficient dataset distillation.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Distributed Search

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Dataset Distillation (DD) compresses large datasets into compact synthetic ones that maintain training performance. However, current methods mainly target sample reduction, with limited consideration of data precision and its impact on efficiency. We propose Quantization-aware Dataset Distillation (QuADD), a unified framework that jointly optimizes dataset compactness and precision under fixed bit budgets. QuADD integrates a differentiable quantization module within the distillation loop, enabling end-to-end co-optimization of synthetic samples and quantization parameters. Guided by the rate-distortion perspective, we empirically analyze how bit allocation between sample count and precision influences learning performance. Our framework supports both uniform and adaptive non-uniform quantization, where the latter learns quantization levels from data to represent information-dense regions better. Experiments on image classification and 3GPP beam management tasks show that QuADD surpasses existing DD and post-quantized baselines in accuracy per bit, establishing a new standard for information-efficient dataset distillation.

Problem

• Dataset Distillation
• Quantization
• Precision
• Compactness
• Bit Budget

Innovation

• Dataset Distillation
• Quantization-aware Optimization
• Rate-Distortion Theory
• Differentiable Quantization
• Information Efficiency