Rethinking Dataset Distillation: Hard Truths about Soft Labels

2026-04-20

AI Summary

Existing dataset distillation methods struggle to significantly outperform random baselines under soft-label settings, often leading to misleading performance evaluations. This work systematically investigates the impact of various labeling strategies—including soft labels, hard labels, and knowledge distillation—on subset quality and model performance, uncovering a performance saturation phenomenon inherent to soft-label distillation. To address this, the authors propose CA2D, an efficient distillation framework tailored for hard-label evaluation. CA2D integrates a computation-aware pruning metric (CAD-Prune), difficulty-aware sampling, and a compute-budget alignment strategy. Evaluated on ImageNet-1K under hard-label constraints, CA2D consistently surpasses both random baselines and state-of-the-art distillation and coreset methods, thereby demonstrating the critical role of data quality in hard-label assessment scenarios.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Despite the perceived success of large-scale dataset distillation (DD) methods, recent evidence finds that simple random image baselines perform on-par with state-of-theart DD methods like SRe2L due to the use of soft labels during downstream model training. This is in contrast with the findings in coreset literature, where high-quality coresets consistently outperform random subsets in the hardlabel (HL) setting. To understand this discrepancy, we perform a detailed scalability analysis to examine the role of data quality under different label regimes, ranging from abundant soft labels (termed as SL+KD regime) to fixed soft labels (SL) and hard labels (HL). Our analysis reveals that high-quality coresets fail to convincingly outperform the random baseline in both SL and SL+KD regimes. In the SL+KD setting, performance further approaches nearoptimal levels relative to the full dataset, regardless of subset size or quality, for a given compute budget. This performance saturation calls into question the widespread practice of using soft labels for model evaluation, where unlike the HL setting, subset quality has negligible influence. A subsequent systematic evaluation of five large-scale and four small-scale DD methods in the HL setting reveals that only RDED reliably outperforms random baselines on ImageNet-1K, but can still lag behind strong coreset methods due to its over-reliance on easy sample patches. Based on this, we introduce CAD-Prune, a compute-aware pruning metric that efficiently identifies samples of optimal difficulty for a given compute budget, and use it to develop CA2D, a compute-aligned DD method, outperforming current DD methods on ImageNet-1K at various IPC settings. Together, our findings uncover many insights into current DD research and establish useful tools to advance dataefficient learning for both coresets and DD.

Problem

• dataset distillation
• soft labels
• hard labels
• coresets
• data efficiency

Innovation

• dataset distillation
• soft labels
• hard labels
• compute-aware pruning
• coreset