Condensing Large-Scale Datasets Directly with Minimal Information Loss

2026-07-01

AI Summary

This work addresses the severe information loss, distribution shift, and limited cross-architecture generalization inherent in existing dataset distillation methods, which rely on multi-stage decoupled pipelines involving dual compression. The authors propose the CIM framework, which for the first time identifies the root cause of information degradation from dual compression and abandons the conventional SQUEEZE-RECOVER-RELABEL paradigm. Instead, CIM employs a metric-driven mechanism to directly align the distributions of original and synthetic data, explicitly minimizing their information gap. Evaluated on ImageNet-1K with IPC=10, the method achieves 48.7% Top-1 accuracy on ResNet-18 within 80 minutes on a single RTX 4090, outperforming state-of-the-art approaches such as NRR-DD and DELT by over 2.6%, while significantly enhancing relabeling efficacy and cross-architecture generalization.

Abstract

Recent advancements in scaling dataset distillation rely heavily on decoupled information extraction pipelines, comprising SQUEEZE, RECOVER, and RELABEL stages. Despite their scalability to large-scale datasets, these methods suffer from prohibitive computational overhead and poor cross-architecture generalization. In this paper, we reveal the root cause of these bottlenecks: the implicit dual-compression process, from data to model and back to images, inherently induces severe information loss. Crucially, we empirically and theoretically demonstrate that this loss creates a distribution shift that fundamentally compromises the widely adopted RELABEL strategy, transforming the pre-trained model into an unreliable labeler that yields sub-optimal labels. To overcome these critical flaws, we propose CIM, a novel, metric-driven framework that abandons the flawed dual-compression paradigm. Instead, CIM explicitly quantifies and minimizes the information gap between the original and synthetic datasets. By directly aligning the data distributions, our approach ensures high-fidelity information condensation and inherently satisfies the prerequisites for effective relabeling. Extensive experiments demonstrate that CIM establishes a new state-of-the-art. Notably, it distills ImageNet-1K at an IPC=10 in merely 80 minutes on a single RTX-4090 GPU, achieving an unprecedented 48.7% Top-1 accuracy on ResNet-18 and significantly outperforming previous SOTA approaches, such as NRR-DD and DELT, by 2.6% and 2.9%, respectively. Our code is available at https://github.com/LINs-lab/CIM.

Problem

• dataset distillation
• information loss
• distribution shift
• relabeling
• cross-architecture generalization

Innovation

• dataset distillation
• information loss
• distribution alignment
• metric-driven framework
• relabeling