🤖 AI Summary
In meta-learning, manually constructing validation sets suffers from poor scalability with increasing classes, difficulty in simultaneously ensuring class balance and label reliability, and dependence on human curation. To address these issues, this paper proposes a utility-driven validation set construction paradigm. We formally define validation set utility along three dimensions: informativeness, necessity (i.e., class balance), and label reliability—introducing the first end-to-end differentiable algorithm, INOLML, optimized for this utility metric. INOLML jointly performs utility-aware validation set selection, self-supervised label correction, and imbalance- and noise-robust meta-training. Extensive experiments on multiple benchmark datasets demonstrate substantial improvements over state-of-the-art methods, establishing new SOTA performance for meta-learning under imbalanced and noisy labels.
📝 Abstract
Meta-learning is an effective method to handle imbalanced and noisy-label learning, but it depends on a validation set containing randomly selected, manually labelled and balanced distributed samples. The random selection and manual labelling and balancing of this validation set is not only sub-optimal for meta-learning, but it also scales poorly with the number of classes. Hence, recent meta-learning papers have proposed ad-hoc heuristics to automatically build and label this validation set, but these heuristics are still sub-optimal for meta-learning. In this paper, we analyse the meta-learning algorithm and propose new criteria to characterise the utility of the validation set, based on: 1) the informativeness of the validation set; 2) the class distribution balance of the set; and 3) the correctness of the labels of the set. Furthermore, we propose a new imbalanced noisy-label meta-learning (INOLML) algorithm that automatically builds a validation set by maximising its utility using the criteria above. Our method shows significant improvements over previous meta-learning approaches and sets the new state-of-the-art on several benchmarks.
