SCORE: Soft Label Compression-Centric Dataset Condensation via Coding Rate Optimization

2025-03-18

AI Summary

Dataset compression (DC) faces a fundamental trade-off between high model performance and prohibitive storage costs—particularly severe in soft-label distillation, where storage overhead often vastly exceeds that of the original dataset. To address this, we propose SCORE, the first DC framework grounded in rate-distortion theory, formulating DC as an information-theoretic minimax optimization that jointly maximizes informativeness, discriminability, and compressibility. We prove the submodularity of the objective, which inherently induces low-rank structure in soft labels and breaks the performance–storage Pareto frontier. SCORE unifies rate optimization, submodular function optimization, and end-to-end distillation. On ImageNet-1K, it achieves 30× soft-label compression: with 10 and 50 images per class (IPC), top-1 accuracy drops by only 5.5% and 2.7%, respectively—substantially outperforming state-of-the-art methods.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Computer Vision: Learning & Optimization for CV
• Machine Learning: Learning on the Edge & Model Compression

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Dataset Condensation (DC) aims to obtain a condensed dataset that allows models trained on the condensed dataset to achieve performance comparable to those trained on the full dataset. Recent DC approaches increasingly focus on encoding knowledge into realistic images with soft labeling, for their scalability to ImageNet-scale datasets and strong capability of cross-domain generalization. However, this strong performance comes at a substantial storage cost which could significantly exceed the storage cost of the original dataset. We argue that the three key properties to alleviate this performance-storage dilemma are informativeness, discriminativeness, and compressibility of the condensed data. Towards this end, this paper proposes a extbf{S}oft label compression-centric dataset condensation framework using extbf{CO}ding extbf{R}at extbf{E} (SCORE). SCORE formulates dataset condensation as a min-max optimization problem, which aims to balance the three key properties from an information-theoretic perspective. In particular, we theoretically demonstrate that our coding rate-inspired objective function is submodular, and its optimization naturally enforces low-rank structure in the soft label set corresponding to each condensed data. Extensive experiments on large-scale datasets, including ImageNet-1K and Tiny-ImageNet, demonstrate that SCORE outperforms existing methods in most cases. Even with 30$ imes$ compression of soft labels, performance decreases by only 5.5% and 2.7% for ImageNet-1K with IPC 10 and 50, respectively. Code will be released upon paper acceptance.

Problem

• Reduces storage cost of condensed datasets
• Balances informativeness, discriminativeness, and compressibility
• Improves performance with soft label compression

Innovation

• Soft label compression via coding rate optimization
• Min-max optimization for dataset condensation
• Low-rank structure in soft label sets