Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models

2026-05-07

AI Summary

Existing dataset distillation methods fail to exploit the closed-form solution of linear probing on frozen pretrained vision models, limiting both efficiency and performance. This work proposes CLP-DD, which for the first time directly integrates closed-form kernel ridge regression into a bilevel distillation framework, eschewing iterative trajectory matching or infinite-width approximations. Synthesized images are optimized in the pretrained feature space via temperature-scaled Softmax cross-entropy, augmented with class anchor learning to enhance discriminability. The method substantially outperforms LGM without DSA and matches or exceeds LGM with DSA on ImageNet-100 and ImageNet-1K, while achieving approximately 14× faster training and using less than one-eighth of the GPU memory.

Abstract

Dataset distillation compresses a large training set into a small synthetic set that preserves downstream training utility. While most existing methods target training networks from scratch, modern visual transfer learning often uses frozen pre-trained encoders followed by lightweight linear probing. Existing distillation methods for this setting either unroll iterative linear-probe updates with trajectory-based gradient matching, or rely on closed-form formulations originally designed for from-scratch training with neural-tangent-kernel (NTK) approximations. Neither route exploits the fact that frozen-feature linear probing admits a closed-form solution determined directly by the pre-trained features themselves, with no infinite-width approximation and no inner-loop trajectory. We propose Closed-Form Linear-Probe Dataset Distillation (CLP-DD), a bilevel formulation that computes the linear probe induced by the synthetic set with a sample-space kernel ridge solver. The synthetic images are then updated by evaluating this induced classifier on real features through a temperature-scaled softmax cross-entropy, where the classifier columns act as learned class anchors in feature space. We further show that the choice of outer objective is decisive: pairing the closed-form inner solver with a standard MSE outer loss substantially underperforms trajectory-based methods, while the discriminative outer loss closes most of the gap. On ImageNet-100 with four pre-trained backbones, CLP-DD substantially improves over LGM without DSA and approaches LGM with DSA at a fraction of the computational cost. On ImageNet-1K, CLP-DD matches or surpasses LGM with DSA on three of four backbones while running roughly $14\times$ faster and using less than one-eighth of the GPU memory.

Problem

• dataset distillation
• linear probing
• pre-trained vision models
• transfer learning
• synthetic dataset

Innovation

• dataset distillation
• linear probing
• closed-form solution
• pre-trained vision models
• kernel ridge regression