Structural Assessment for Understanding and Guiding Dataset Distillation in Discrete Token Space

2026-06-19

AI Summary

This work addresses the limited understanding of semantic structure in existing dataset distillation methods. The authors propose parsing distilled samples into discrete visual tokens using a visual tokenizer and introduce, for the first time, a structural score to quantify the adequacy and balance of their semantic composition. They demonstrate a strong correlation between this score and model performance. By integrating the structural score into the diffusion-based distillation process, the method effectively guides the generation of high-quality distilled data. Experiments show that samples with high structural scores significantly improve model validation performance, and remarkably, even those deviating from the original data distribution can maintain or enhance distillation efficacy. This establishes a novel, semantics-aware criterion for dataset distillation.

Abstract

Dataset distillation (DD) has proven to reduce training cost while preserving accuracy. While promising, the factors that make one distilled dataset more effective than another remain poorly understood. In this work, we investigate this question through the lens of discrete visual tokenizers. Whereas many prior DD efforts emphasize matching global data distributions, we suggest that the effectiveness depends on which semantic concepts are captured and how they are composed. Discrete visual tokenizers provide a finite vocabulary that enables direct statistical analysis of such compositional structure. Through quantitative analysis of token-level statistics, we introduce the structural score to measure the adequacy of token compositions. We observe that distilled datasets with balanced token composition yield higher validation performance. On the other hand, divergence from the original data does not necessarily harm performance. We further show that samples with high structural scores in the discrete token space can effectively guide diffusion-based DD. Our findings highlight the importance of token composition in dataset effectiveness, offering a principled complement to distributional similarity considerations in DD.

Problem

• dataset distillation
• discrete token space
• structural assessment
• token composition
• semantic concepts

Innovation

• dataset distillation
• discrete token space
• structural score
• token composition
• diffusion-based distillation