Generative Dataset Distillation using Min-Max Diffusion Model

2025-03-24

AI Summary

This work addresses the efficiency–fidelity trade-off in generative dataset distillation using diffusion models. We propose an efficient and high-fidelity proxy data synthesis method. Our approach features: (1) a Min-Max loss function that jointly optimizes feature-space distribution matching and discriminative robustness of synthesized data; and (2) Diffusion Step Reduction—a novel, gradient-driven sampling step pruning strategy that significantly accelerates generation without compromising fidelity. The method integrates adversarial optimization, differentiable step control, and a closed-loop distillation evaluation during training. Evaluated on the ECCV 2024 First Dataset Distillation Challenge (generation track), our method ranked second. It achieves substantial improvements in few-shot downstream task accuracy and reduces per-image generation latency by 37%–52%.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

In this paper, we address the problem of generative dataset distillation that utilizes generative models to synthesize images. The generator may produce any number of images under a preserved evaluation time. In this work, we leverage the popular diffusion model as the generator to compute a surrogate dataset, boosted by a min-max loss to control the dataset's diversity and representativeness during training. However, the diffusion model is time-consuming when generating images, as it requires an iterative generation process. We observe a critical trade-off between the number of image samples and the image quality controlled by the diffusion steps and propose Diffusion Step Reduction to achieve optimal performance. This paper details our comprehensive method and its performance. Our model achieved $2^{nd}$ place in the generative track of href{https://www.dd-challenge.com/#/}{The First Dataset Distillation Challenge of ECCV2024}, demonstrating its superior performance.

Problem

• Generative dataset distillation using diffusion models
• Balancing image quality and sample quantity efficiently
• Optimizing diffusion steps for performance and speed

Innovation

• Uses diffusion model for dataset distillation
• Implements min-max loss for diversity control
• Proposes Diffusion Step Reduction optimization