Information-Guided Diffusion Sampling for Dataset Distillation

2025-07-06

AI Summary

Dataset distillation under low images-per-class (IPC) settings often suffers from insufficient sample diversity generated by diffusion models, leading to loss of both class prototypes and intra-class contextual structure. To address this, we propose an information-guided diffusion sampling framework that introduces variational mutual information estimation into distillation for the first time. Our method formulates an IPC-adaptive information maximization objective: maximize (I(X;Y) + eta H(X|Y)), jointly preserving class discriminability (prototypes) and intra-class structural coherence (context). Crucially, it requires no architectural modifications—instead, it achieves data-efficient distillation via information-driven reweighting during the sampling stage. Evaluated on Tiny ImageNet and an ImageNet subset, our approach consistently outperforms state-of-the-art methods across 1–5 IPC settings; notably, it achieves a 4.2% absolute Top-1 accuracy gain at 1 IPC. These results demonstrate the effectiveness and generalizability of information-guided sampling for extreme low-resource distillation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Information Theory
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Web Mining and Content Analysis: Content-based information diffusion
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Dataset distillation aims to create a compact dataset that retains essential information while maintaining model performance. Diffusion models (DMs) have shown promise for this task but struggle in low images-per-class (IPC) settings, where generated samples lack diversity. In this paper, we address this issue from an information-theoretic perspective by identifying two key types of information that a distilled dataset must preserve: ($i$) prototype information $mathrm{I}(X;Y)$, which captures label-relevant features; and ($ii$) contextual information $mathrm{H}(X | Y)$, which preserves intra-class variability. Here, $(X,Y)$ represents the pair of random variables corresponding to the input data and its ground truth label, respectively. Observing that the required contextual information scales with IPC, we propose maximizing $mathrm{I}(X;Y) + βmathrm{H}(X | Y)$ during the DM sampling process, where $β$ is IPC-dependent. Since directly computing $mathrm{I}(X;Y)$ and $mathrm{H}(X | Y)$ is intractable, we develop variational estimations to tightly lower-bound these quantities via a data-driven approach. Our approach, information-guided diffusion sampling (IGDS), seamlessly integrates with diffusion models and improves dataset distillation across all IPC settings. Experiments on Tiny ImageNet and ImageNet subsets show that IGDS significantly outperforms existing methods, particularly in low-IPC regimes. The code will be released upon acceptance.

Problem

• Improves dataset distillation in low-IPC settings using diffusion models
• Preserves prototype and contextual information for better sample diversity
• Proposes variational estimations to optimize information retention during sampling

Innovation

• Maximizes prototype and contextual information
• Uses variational estimations for intractable computations
• Integrates information-guided diffusion sampling (IGDS)