Cost-Effective Model Evaluation with Meta-Learning

2026-05-22

AI Summary

This work addresses the challenge of efficiently and reliably evaluating newly released machine learning models on unlabeled data without incurring costly annotations or repeated fine-tuning. The authors propose MetaEvaluator, the first model-agnostic, label-free, and training-free evaluation framework that eliminates the need for per-model retraining. Leveraging meta-learning, MetaEvaluator learns a transferable evaluator initialization from a pool of reference models, enabling rapid unsupervised performance estimation for models of unseen architectures and modalities. Extensive experiments demonstrate that MetaEvaluator consistently and accurately predicts model performance across diverse datasets and model types, substantially reducing evaluation costs and facilitating large-scale benchmarking on unlabeled data.

Abstract

The rapid growth of machine learning has produced an ever-expanding ecosystem of models, making it increasingly challenging to verify the reliability of newly released models on unseen, unlabeled data. Conventional evaluation pipelines depend on expensive annotation, repeated fine-tuning, or narrow assumptions that fail to transfer across model families. We present MetaEvaluator, a cost-effective, model-agnostic framework for rapid, label-free assessment of unseen models spanning diverse architectures and modalities. MetaEvaluator leverages meta-learning over a pool of reference models to obtain a transferable initialization, enabling accurate evaluation of new models while amortizing cost across the pool and removing the need for per-model retraining. To the best of our knowledge, this is the first model-agnostic framework capable of evaluating new models on entirely unlabeled datasets. Extensive experiments show that MetaEvaluator produces stable and accurate performance estimates at substantially reduced cost compared to conventional approaches, making scalable benchmarking of emerging models on unlabeled data practical.

Problem

• model evaluation
• unlabeled data
• cost-effective
• model-agnostic
• meta-learning

Innovation

• meta-learning
• model evaluation
• label-free
• model-agnostic
• cost-effective