Towards Principled Dataset Distillation: A Spectral Distribution Perspective

2026-03-02

AI Summary

Existing dataset distillation methods suffer significant performance degradation in long-tailed scenarios, primarily due to heuristic distribution alignment strategies and uniform treatment of imbalanced classes. This work proposes a Class-aware Spectral Distribution Matching (CSDM) framework, which formalizes the distribution alignment problem from a spectral perspective for the first time. By mapping samples into the frequency domain via kernel functions, CSDM constructs a Spectral Distribution Distance (SDD) and leverages magnitude–phase decomposition to enable class-adaptive enhancement of tail classes. On CIFAR-10-LT, with only 10 synthesized images per class, CSDM improves performance by 14.0% over the current state-of-the-art method. Moreover, when the number of tail-class images is reduced from 500 to 25, performance declines by merely 5.7%, demonstrating remarkable stability and effectiveness.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Search and Optimization: Distributed Search

Application Category

• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Dataset distillation (DD) aims to compress large-scale datasets into compact synthetic counterparts for efficient model training. However, existing DD methods exhibit substantial performance degradation on long-tailed datasets. We identify two fundamental challenges: heuristic design choices for distribution discrepancy measure and uniform treatment of imbalanced classes. To address these limitations, we propose Class-Aware Spectral Distribution Matching (CSDM), which reformulates distribution alignment via the spectrum of a well-behaved kernel function. This technique maps the original samples into frequency space, resulting in the Spectral Distribution Distance (SDD). To mitigate class imbalance, we exploit the unified form of SDD to perform amplitude-phase decomposition, which adaptively prioritizes the realism in tail classes. On CIFAR-10-LT, with 10 images per class, CSDM achieves a 14.0% improvement over state-of-the-art DD methods, with only a 5.7% performance drop when the number of images in tail classes decreases from 500 to 25, demonstrating strong stability on long-tailed data.

Problem

• dataset distillation
• long-tailed datasets
• class imbalance
• distribution discrepancy
• spectral distribution

Innovation

• Dataset Distillation
• Spectral Distribution
• Long-Tailed Learning
• Class Imbalance
• Kernel Spectrum