PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents

2026-08-25

AI Summary

为解决LLM代理在多工具并行执行时的资源管理问题，引入PeakBench基准测试，通过分离逻辑规划与物理调度评估方法，提高资源利用效率。

Abstract

LLM agents increasingly solve tasks by invoking multiple tools, where parallel execution is essential for low latency but difficult to manage safely. Existing agent benchmarks primarily evaluate tool selection, argument generation, and end-to-end success under mostly serial execution, largely overlooking valid parallelization and resource-constrained scheduling. This missing scheduling dimension creates a practical failure mode: serial execution is safe but slow, while resource-agnostic parallel execution is fast but prone to avoidable resource overflows. To address this gap, we introduce PeakBench, a benchmark of executable multi-tool workflows with execution-grounded dependency annotations and measured resource profiles. A central challenge in evaluating such workflows is attribution: failures and inefficiencies may arise from incorrect dependency planning, poor resource-constrained scheduling, or both. PeakBench addresses this challenge with a two-part evaluation framework that disentangles logical planning from physical scheduling, with dedicated metrics for each dimension. Using this framework, we show that strong logical planning does not reliably translate into safe or efficient execution under resource constraints. We further show that exposing resource information can reduce avoidable overflows and improve resource utilization, making PeakBench a useful testbed for diagnosing resource-aware agent behavior. Code is available at https://github.com/Czzzk/Staggering-the-Peaks.

Problem

• Resource-Aware
• Parallel Execution
• Scheduling
• LLM Agents
• Tool Invocation

Innovation

• Resource-Aware Scheduling
• Dependency Planning
• Multi-Tool Workflows
• Benchmarking