Spectral Gradient Surgery for Domain-Generalizable Dataset Distillation

2026-05-12

AI Summary

This work addresses the limited out-of-distribution generalization of existing dataset distillation methods and their poor adaptability to synthetic data lacking diversity. To overcome these limitations, we propose Domain-Generalizable Dataset Distillation (DGDD), a novel paradigm that explicitly incorporates domain generalization objectives into the distillation process. DGDD introduces Spectral Gradient Surgery (SGS) to disentangle class-discriminative from domain-specific information, leverages cross-domain gradient consistency to identify shared features, and employs a dual-branch gradient update mechanism to simultaneously enhance discriminability and diversity. Built upon a distribution matching framework, DGDD significantly improves out-of-distribution generalization across multiple benchmark scales and is readily plug-and-play compatible with existing distillation approaches, achieving both efficiency and effectiveness.

Abstract

Dataset Distillation (DD) synthesizes a compact synthetic dataset that preserves the training utility of a full dataset. However, its standard formulation assumes that test data follow the same distribution as training data, an assumption that rarely holds in practice. A straightforward extension-applying post-hoc Domain Generalization (DG) techniques to distilled data-is ill-suited because existing DG methods rely on the natural diversity of real datasets, which compact synthetic sets inherently lack, while also incurring substantial augmentation overhead that conflicts with the efficiency objective of dataset distillation. To address this limitation, we introduce Domain Generalizable Dataset Distillation (DGDD), a new problem setting that explicitly targets out-of-distribution (OOD) generalization of distilled datasets. We study this problem through a widely adopted DD baseline of Distribution Matching (DM). We attribute the OOD vulnerability of DM to the entanglement of class-discriminative and domain-specific information within the compressed synthetic set, and propose Spectral Gradient Surgery (SGS) to disentangle the two. The key insight of SGS is that cross-domain agreement among domain-wise gradients in the spectral domain reveals which gradient components are shared across source domains-and are therefore class-discriminative-and which are domain-specific. Based on this observation, SGS augments the standard DM update with two complementary gradients: one that reinforces cross-domain shared components and another that explicitly promotes diversity within the distilled dataset. Extensive experiments on diverse-scale benchmarks demonstrate that SGS substantially improves OOD generalization while remaining plug-and-play compatible with existing DM methods.

Problem

• Dataset Distillation
• Domain Generalization
• Out-of-Distribution Generalization
• Synthetic Dataset
• Distribution Shift

Innovation

• Spectral Gradient Surgery
• Domain Generalizable Dataset Distillation
• Out-of-Distribution Generalization
• Dataset Distillation
• Distribution Matching