Not All Samples Should Be Utilized Equally: Towards Understanding and Improving Dataset Distillation

2024-08-22

arXiv.org

AI Summary

Dataset distillation (DD) suffers from performance degradation under low images-per-class (IPC) regimes and lacks a theoretical understanding of how sample difficulty affects distillation. This paper presents the first unified analysis of matching-based DD methods from the perspective of sample difficulty, revealing their implicit bias toward easily learnable samples. We establish a theoretical framework for quantifying sample difficulty based on gradient norm magnitude. Building upon this insight, we propose Sample Difficulty Correction (SDC), a plug-and-play mechanism that explicitly steers the distillation process to prioritize synthesizing easily learnable samples. SDC integrates gradient-norm-based difficulty measurement, an extended neural scaling law, and optimized matching loss. Evaluated across six benchmarks and seven baseline methods, SDC consistently improves distilled dataset accuracy—yielding average gains of 2.1–5.7 percentage points under low-IPC settings. Our work establishes an interpretable, reusable, difficulty-aware paradigm for dataset distillation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Security and Privacy: Large-scale security measurements

Abstract

Dataset Distillation (DD) aims to synthesize a small dataset capable of performing comparably to the original dataset. Despite the success of numerous DD methods, theoretical exploration of this area remains unaddressed. In this paper, we take an initial step towards understanding various matching-based DD methods from the perspective of sample difficulty. We begin by empirically examining sample difficulty, measured by gradient norm, and observe that different matching-based methods roughly correspond to specific difficulty tendencies. We then extend the neural scaling laws of data pruning to DD to theoretically explain these matching-based methods. Our findings suggest that prioritizing the synthesis of easier samples from the original dataset can enhance the quality of distilled datasets, especially in low IPC (image-per-class) settings. Based on our empirical observations and theoretical analysis, we introduce the Sample Difficulty Correction (SDC) approach, designed to predominantly generate easier samples to achieve higher dataset quality. Our SDC can be seamlessly integrated into existing methods as a plugin with minimal code adjustments. Experimental results demonstrate that adding SDC generates higher-quality distilled datasets across 7 distillation methods and 6 datasets.

Problem

• Understanding dataset distillation through sample difficulty analysis
• Theoretical explanation of matching-based methods via neural scaling laws
• Improving distilled dataset quality by prioritizing easier samples

Innovation

• Prioritize easier samples synthesis
• Sample Difficulty Correction approach
• Seamless plugin for existing methods