DIVER:Diving Deeper into Distilled Data via Expressive Semantic Recovery

2026-05-12

AI Summary

Existing dataset distillation methods predominantly adopt a single-stage paradigm, which tends to overfit specific model architectures, thereby limiting semantic expressiveness and impairing cross-architecture generalization. To address this, this work proposes DIVER, a novel two-stage distillation framework that introduces an innovative semantic inheritance–guidance–fusion mechanism. During the reverse generation process of a pretrained diffusion model, DIVER dynamically injects high-level semantic guidance to effectively suppress architecture-specific noise and semantic artifacts. This approach substantially enhances both semantic fidelity and generalization of the distilled data. Notably, DIVER achieves inference efficiency comparable to the original DiT while requiring only 4GB of GPU memory on ImageNet at 256×256 resolution.

Abstract

Dataset distillation aims to synthesize a compact proxy dataset that is unreadable or non-raw from the original dataset for privacy protection and highly efficient learning. However, previous approaches typically adopt a single-stage distillation paradigm, which suffers from learning specific patterns that overfit on a prior architecture, consequently suppressing the expression of semantics and leading to performance degradation across heterogeneous architectures. To address this issue, we propose a novel dual-stage distillation framework called ${\textbf{DIVER}}$, which leverages the pre-trained diffusion model to dive deeper into $\textbf{DI}$stilled data $\textbf{V}$ia $\textbf{E}$xpressive semantic $\textbf{R}$ecovery, an entire process of semantic inheritance, guidance, and fusion. Semantic inheritance distills high-level semantics of abstract distilled images into the latent space to filter out architecture-specific ``noise" and retain the intrinsic semantics. Furthermore, semantic guidance improves the preservation of the original semantics by directing the reverse procedure. Finally, semantic fusion is designed to provide semantic guidance only during the concrete phase of the reverse process, preventing semantic ambiguity and artifacts while maintaining the guidance information. Extensive experiments validate the effectiveness and efficiency of DIVER in improving classical distillation techniques and significantly improving cross-architecture generalization, requiring processing time comparable to raw DiT on ImageNet (256$\times$256) with only 4 GB of GPU memory usage. Code is available: https://github.com/einsteinxia/DIVER.

Problem

• dataset distillation
• cross-architecture generalization
• semantic expression
• overfitting
• privacy-preserving learning

Innovation

• dataset distillation
• diffusion model
• semantic recovery
• cross-architecture generalization
• dual-stage distillation