PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Calling

2026-08-09

AI Summary

Existing tool-calling evaluation benchmarks suffer from insufficient coverage of long-tail scenarios, a lack of adversarial negative samples, and reliance on large language model (LLM) annotations未经 execution verification, hindering fine-grained error attribution. This work proposes a two-stage diagnostic benchmark construction framework: first decoupling candidate tool-call generation from deterministic validation based on actual API execution, and then generating multi-difficulty queries—including adversarial negatives targeting omission, hallucination, and parameter errors—through decomposition along plugin capability, intent, and boundary dimensions, iteratively refined until convergence. By integrating execution-verified labels and an LLM-as-judge mechanism, the approach overcomes the limitations of conventional benchmarks that rely solely on end-to-end accuracy. Evaluations across five major model families demonstrate that the proposed benchmark precisely characterizes error profiles, reveals performance disparities across difficulty levels and error types, and achieves high agreement between LLM judgments and human annotations.

Abstract

Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents. Current benchmarks face three structural limitations: data distributions that follow a power law leave rare scenarios underrepresented; the absence of adversarial hard negatives obscures performance differences across models; and annotation pipelines depend on LLM judgments that have not been validated through execution. In this paper, we introduce PluginEval, a benchmark constructed through a two-stage framework that systematically mitigates these limitations. First, we formulate tool routing as a sequence of three decisions and separate generation from verification. LLMs propose candidate calls, while deterministic validation and real API execution provide reliable quality signals. Second, we decompose each plugin by capability, intent, and boundary to identify trigger and exclusion scenarios. We then generate queries at different difficulty levels to fill coverage gaps, including adversarial negatives targeting three failure modes, and return them to the first stage for annotation. This process creates a closed loop that iterates until coverage converges. For evaluation, we move beyond aggregate accuracy. An LLM judge anchored to gold annotations classifies failures as missed calls, spurious calls, or parameter errors, producing a detailed error profile for each model. We evaluate five model families, including proprietary models and models with open weights, analyze their performance across difficulty levels and error categories, and validate the judge through agreement with human annotations.

Problem

• tool routing
• benchmark
• error attribution
• adversarial negatives
• LLM evaluation

Innovation

• function calling
• tool routing
• adversarial evaluation
• error attribution
• closed-loop benchmarking