Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets?

2026-06-16

AI Summary

This study addresses the lack of a standardized evaluation protocol in dataset distillation research, which has hindered objective comparisons between distilled datasets and real-data baselines such as coreset methods. Under a unified experimental setup, the authors conduct the first systematic comparison of seven state-of-the-art distillation techniques against three coreset selection strategies across ImageNet-1K, ImageNet100, and ImageNette, employing both standard empirical risk minimization (ERM) and single/multi-teacher training protocols. Comprehensive evaluations along dimensions of accuracy, representativeness, diversity, and distributional coverage reveal that current distillation approaches do not consistently outperform—and often underperform—coreset methods on large-scale datasets, despite incurring substantially higher computational costs. Notably, coresets demonstrate superior coverage of the original data distribution.

Abstract

Dataset distillation (DD) has emerged as a prominent approach in data centric machine learning, aiming to synthesize compact training sets for efficient training by compressing the information in large datasets into a small number of synthetic samples. However, DD methods are often evaluated under inconsistent evaluation protocols, ranging from standard ERM to single/multi-teacher supervision, making it difficult to isolate the effectiveness of distilled data from evaluation. Moreover, many prior methods claim that DD outperforms data pruning approaches such as coreset selection (CS), based on the assumption that restricting condensed datasets to subsets of real samples fundamentally limits their expressiveness. In this work, we critically evaluate DD methods through large-scale experiments using standardized datasets and evaluation protocols to assess their intrinsic effectiveness. We benchmark seven state-of-the-art (SOTA) DD methods on ImageNet-1K, ImageNet100, and ImageNette, using three widely adopted training protocols against three CS strategies. Our results show that while some DD methods fail to outperform even simple random subsets, the SOTA DD approaches are comparable to or worse than coresets on large-scale datasets and incur a substantially higher cost for construction. Beyond accuracy, we also evaluate the representativeness, diversity, and quality of condensed sets, and find that coresets consistently achieve better coverage of the original data distribution. These findings highlight the limited practical advantages of current DD methods and show that coresets remain competitive and are often a more computationally efficient alternative for data-centric learning.

Problem

• dataset distillation
• coreset selection
• evaluation protocol
• data efficiency
• representativeness

Innovation

• dataset distillation
• coreset selection
• data-centric learning
• evaluation protocol
• distribution coverage