Towards Direct Evaluation of Harness Optimizers via Priority Ranking

2026-05-21

AI Summary

Existing evaluation methods assess optimizer effectiveness only indirectly through the final performance of the target agent, making it difficult to evaluate the quality of individual optimization decisions. This work proposes a prioritization-based approach that requires optimizers to rank components—such as tools—according to their potential performance gain, enabling direct, step-level evaluation without costly replay or manual inspection. Leveraging the Shor dataset, which comprises 182 human-validated cross-domain scenarios, we introduce the first low-cost, scalable benchmark task tailored for harness optimizers. Experimental results demonstrate a strong correlation between an optimizer’s performance on this ranking task and its ultimate effectiveness in real multi-step optimization, thereby validating the proposed method as a reliable predictive metric.

Abstract

Harness optimization enables automated agent creation by having an optimizer agent iteratively update the harness of target agents. Despite its success, current studies evaluate optimizers solely by observing target agents' performance gains. This indirect end-improvement evaluation neglects optimizers' actions at intermediate steps, which are often erroneous and hinder agent performance. Therefore, it is unclear whether harness optimization is driven by optimizers' informed update actions or simply trial-and-error. This necessitates direct evaluation of harness optimizers. However, evaluating harness optimizers directly is non-trivial and costly due to the lack of oracle harnesses. To address this, we present a simple, low-cost design to directly evaluate them, namely priority ranking. By asking harness optimizers to rank components (e.g., tools) in a given harness by their potential to improve/hinder agent performance when updated, our design quantifies optimizer ability at the step level without expensive rollouts or manual examination. More importantly, optimizers' ranking performance correlates with their ability to improve agents in actual multi-step harness optimization, establishing priority ranking as a reliable predictor of optimization ability. Priority ranking is enabled by Shor, a collection of 182 human-verified optimization scenarios spanning across domains, designs, and time stages. Codes and data can be found at https://github.com/k59118/Harness_Optimizer_Evaluation.

Problem

• harness optimization
• optimizer evaluation
• priority ranking
• agent performance
• direct evaluation

Innovation

• priority ranking
• harness optimization
• direct evaluation
• optimizer assessment
• Shor dataset