Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation

2026-07-07

AI Summary

This work addresses the critical challenge in few-shot knowledge distillation of efficiently selecting an extremely small yet representative subset of samples to enhance student model performance. The authors propose a minimalist coreset selection method that computes the medoid for each class and selects the sample closest to this medoid as the distilled data. By eschewing complex sampling strategies and relying solely on intra-class geometric structure, the approach achieves remarkable simplicity while significantly outperforming random sampling and existing coreset methods. Extensive experiments demonstrate consistent improvements across four image classification benchmarks and three teacher–student architectures—including both CNNs and Transformers—establishing the method as a strong standard baseline for few-shot knowledge distillation.

Abstract

Coreset selection aims to identify a small and highly representative subset of a massive dataset for efficient model training. The problem remains challenging even in the few-shot knowledge distillation (KD) setup, where a full-scale pre-trained teacher informs the student network. Typical sample selection strategies often struggle to surpass the random selection baseline. In this paper, we showcase few-medoids, an embarrassingly simple coreset selection strategy that chooses the samples closest to the centroid (average image) of each class. We present extensive KD experiments on four datasets, covering a wide range of image classification problems, and three teacher-student model pairs, comprising both convolutional and transformer networks. Although the proposed method is embarrassingly simple, our empirical results indicate that few-medoids is able to consistently surpass the random selection baseline, as well as the other coreset selection strategies. We therefore consider that few-medoids can be used as a drop-in replacement for commonly-used baselines (e.g. herding or k-center Greedy), in future research on coreset selection. To reproduce the reported results, we publicly release our code at https://github.com/CemilAndreiDilmac/Few-Shot-KD-Coreset.

Problem

• coreset selection
• few-shot knowledge distillation
• sample selection
• representative subset
• efficient model training

Innovation

• Few-Medoids
• Coreset Selection
• Few-Shot Knowledge Distillation
• Centroid-based Sampling
• Model Compression