Rectified Decoupled Dataset Distillation: A Closer Look for Fair and Comprehensive Evaluation

2025-09-23

AI Summary

Decoupled dataset distillation has long suffered from inconsistent post-evaluation protocols, undermining fair and rigorous comparison across methods. Method: We propose RD³, the first standardized benchmark with a strict, reproducible evaluation protocol. It systematically investigates the impacts of stochastic augmentation, round-wise soft-labeling, and multi-scenario post-evaluation on distillation performance. Through controlled ablation studies, we isolate evaluation-induced biases from intrinsic synthetic-data quality differences. Contribution/Results: We demonstrate that performance disparities among existing methods stem primarily from evaluation artifacts—not genuine data fidelity gaps. Leveraging these insights, we derive a robust, efficient distillation strategy invariant to evaluation setup variations. This work is the first to disentangle and clarify the true source of performance gains in decoupled distillation, establishing a fair, reproducible evaluation paradigm grounded in real-data efficacy—thereby redirecting research focus toward meaningful generalization rather than protocol-specific overfitting.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Dataset distillation aims to generate compact synthetic datasets that enable models trained on them to achieve performance comparable to those trained on full real datasets, while substantially reducing storage and computational costs. Early bi-level optimization methods (e.g., MTT) have shown promising results on small-scale datasets, but their scalability is limited by high computational overhead. To address this limitation, recent decoupled dataset distillation methods (e.g., SRe$^2$L) separate the teacher model pre-training from the synthetic data generation process. These methods also introduce random data augmentation and epoch-wise soft labels during the post-evaluation phase to improve performance and generalization. However, existing decoupled distillation methods suffer from inconsistent post-evaluation protocols, which hinders progress in the field. In this work, we propose Rectified Decoupled Dataset Distillation (RD$^3$), and systematically investigate how different post-evaluation settings affect test accuracy. We further examine whether the reported performance differences across existing methods reflect true methodological advances or stem from discrepancies in evaluation procedures. Our analysis reveals that much of the performance variation can be attributed to inconsistent evaluation rather than differences in the intrinsic quality of the synthetic data. In addition, we identify general strategies that improve the effectiveness of distilled datasets across settings. By establishing a standardized benchmark and rigorous evaluation protocol, RD$^3$ provides a foundation for fair and reproducible comparisons in future dataset distillation research.

Problem

• Inconsistent post-evaluation protocols hinder progress in dataset distillation
• Performance differences may stem from evaluation discrepancies rather than methodological advances
• Lack of standardized benchmark for fair comparison of decoupled distillation methods

Innovation

• Decouples teacher training from synthetic data generation
• Systematically analyzes post-evaluation protocol impacts
• Establishes standardized benchmark for fair comparisons