PACE: A Proxy for Agentic Capability Evaluation

2026-07-02

AI Summary

This work addresses the high cost and procedural complexity of evaluating large agent models, which severely hinder development and iteration efficiency. The authors propose the PACE framework, which constructs a lightweight proxy benchmark by selecting critical instances from non-agent atomic capability evaluations through a hybrid strategy combining task-relevant local selection and global informativeness-based selection. This approach enables low-cost, high-fidelity prediction of model performance on complex multi-agent tasks. Experimental results across 14 models demonstrate that PACE achieves an average absolute error below 4%, Spearman correlation exceeding 0.80, and model ranking accuracy of 85%, all while using less than 1% of the original evaluation cost—marking the first method to enable highly accurate and efficient prediction for multi-agent benchmarks.

Abstract

Evaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure. A single evaluation can cost thousands of dollars and take days to complete. In contrast, non-agentic LLM benchmarks that test individual capabilities (e.g., reasoning, code generation) are fast and cheap to run. In this paper, we investigate whether performance on expensive agentic benchmarks can be accurately predicted by the performance on a small, carefully selected subset of atomic evaluation instances. We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations whose aggregate scores most reliably predict model performances on agentic benchmarks. Given a pool of candidate instances spanning atomic capabilities, PACE fits a regression that maps a model's scores on a compact subset of source instances to its score on the target agentic benchmark. The subset itself is curated by combining two complementary instance-selection strategies, target-relevance local selection and globally informative global selection. We apply PACE to the 4 target agentic benchmarks in this paper, which yields PACE-Bench, the concrete proxy benchmark that we evaluate in the paper. Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost. We further analyze the selected proxy instances, revealing which skills each agentic benchmark uniquely demands. PACE enables practitioners to obtain reliable estimates of agentic performance during model development, selection, and routing, without the overhead of full agent evaluation.

Problem

• agentic evaluation
• proxy benchmark
• LLM agents
• performance prediction
• evaluation cost

Innovation

• proxy benchmark
• agentic evaluation
• instance selection
• regression-based prediction
• large language model agents