AgentMeter: Evaluating Model-CLI Matching for CLI-Based Local Task-Solving Agents

2026-06-19

AI Summary

Existing evaluations focus solely on task success rates, overlooking the critical impact of command-line interfaces (CLIs) on the performance of local agents and thus failing to accurately assess the joint deployment effectiveness of models and CLIs. This work proposes AgentMeter, a novel benchmark that treats the model–CLI pair as a unified deployment unit for holistic evaluation, introducing a cost-aware scoring mechanism—AgentMeter Score (AMS)—calibrated by task-effort tiers. Experiments on the full Benchmark90 suite and its Core30 subset demonstrate that AMS effectively quantifies the overall efficacy of 24 distinct model–CLI configurations and identifies optimal setups under varying optimization objectives. The top-performing configuration exhibits consistent results across Benchmark90, with AMS showing high internal consistency (Spearman correlation of 0.765 and MAE of 0.0383).

Abstract

LLM agents increasingly solve local tasks through command-line and CLI-based harness interfaces, including code editing, repository inspection, data analysis, and file workflows. Existing evaluations often emphasize task success, but deployed local agents are not models alone: the CLI mediates prompts, context replay, tool outputs, file access, terminal observations, and stopping behavior. As a result, the same model can produce different success, token, and cost profiles under different CLIs. We introduce AGENTMETER, a benchmark for evaluating model-CLI matching in CLI-mediated local task-solving agents, together with AgentMeter Score (AMS), a success-anchored, cost-aware metric over calibrated task-effort tiers. AgentMeter uses Benchmark90 as the full validation set and Core30 as a lower-cost subset for expanded comparison across 24 complete model-CLI configurations. On Core30, common deployment criteria select different configurations: highest Pass/30 selects GLM-5.1 with qwen-coder, lowest Tok./Pass selects GPT-5.3-Codex with kimi-cli, lowest billable USD/Pass selects Qwen3.6+ with Codex, while highest AMS selects Qwen3.6+ with kimi-cli. Benchmark90 validation preserves the Top-1 configuration and Top-3 set, with Spearman correlation 0.765, Kendall correlation 0.567, and AMS MAE 0.0383. These results show that model choice and CLI choice should not be decoupled, and that model-CLI configurations should be evaluated as the deployed unit.

Problem

• CLI-based agents
• model-CLI matching
• local task-solving
• evaluation benchmark
• agent performance

Innovation

• model-CLI matching
• AgentMeter
• cost-aware evaluation
• CLI-mediated agents
• AgentMeter Score