How Benchmark Prediction from Fewer Data Misses the Mark

2025-06-09

AI Summary

This work addresses benchmark prediction for efficient large language model (LLM) evaluation—estimating a model’s full-benchmark performance from few-shot samples. We find that existing methods critically rely on known model capability distributions and consistently fail when extrapolating to frontier, unseen strong models; remarkably, a random-sampling-plus-regression baseline outperforms most state-of-the-art approaches. To overcome this limitation, we propose a novel framework based on augmented inverse propensity weighting (IPW), the first method to robustly surpass random-mean prediction under cross-capability extrapolation. Extensive evaluation across 19 benchmarks and 11 methods demonstrates consistent effectiveness—but modest gains—revealing a fundamental limitation: benchmark prediction is inherently constrained in regions of unknown model capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language model (LLM) evaluation is increasingly costly, prompting interest in methods that speed up evaluation by shrinking benchmark datasets. Benchmark prediction (also called efficient LLM evaluation) aims to select a small subset of evaluation points and predict overall benchmark performance from that subset. In this paper, we systematically assess the strengths and limitations of 11 benchmark prediction methods across 19 diverse benchmarks. First, we identify a highly competitive baseline: Take a random sample and fit a regression model on the sample to predict missing entries. Outperforming most existing methods, this baseline challenges the assumption that careful subset selection is necessary for benchmark prediction. Second, we discover that all existing methods crucially depend on model similarity. They work best when interpolating scores among similar models. The effectiveness of benchmark prediction sharply declines when new models have higher accuracy than previously seen models. In this setting of extrapolation, none of the previous methods consistently beat a simple average over random samples. To improve over the sample average, we introduce a new method inspired by augmented inverse propensity weighting. This method consistently outperforms the random sample average even for extrapolation. However, its performance still relies on model similarity and the gains are modest in general. This shows that benchmark prediction fails just when it is most needed: at the evaluation frontier, where the goal is to evaluate new models of unknown capabilities.

Problem

• Evaluating LLMs with fewer data points accurately
• Assessing benchmark prediction methods across diverse benchmarks
• Improving prediction for new high-accuracy models

Innovation

• Random sampling with regression for benchmark prediction
• Augmented inverse propensity weighting improves extrapolation
• Model similarity crucial for benchmark prediction effectiveness