PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems

2026-06-21

AI Summary

Existing benchmarks struggle to evaluate large language models’ long-horizon planning capabilities under conditions of limited tool visibility and dynamic disruptions. This work proposes the first interactive evaluation benchmark featuring an optional blocking mechanism, encompassing 327 retail tasks and 1,665 tools. By dynamically blocking tool access to simulate scenarios of tool unavailability or failure, and integrating task flows derived from large-scale API call trajectories with intermediate evidence-based reasoning, the benchmark systematically assesses agents’ abilities in iterative retrieval and path replanning. Experimental results reveal that even the strongest model tested (GPT-5.4) achieves only 51.90% accuracy without blocking, which sharply declines to 11.36% under severe blocking, exposing significant deficiencies in implicit failure detection and recovery over extended planning horizons.

Abstract

LLM agents increasingly operate in large tool ecosystems, where real-world tasks require discovering relevant tools, inferring implicit sub-goals, and adapting to dynamic environments over long horizons. However, existing benchmarks rarely evaluate planning under retrieval-limited tool visibility. To address this gap, we introduce PlanBench-XL, an interactive benchmark of 327 retail tasks over 1,665 tools that tests whether agents can iteratively retrieve usable tools, invoke them to uncover intermediate evidence for subsequent calls toward the final goal. PlanBench-XL further features an optional blocking mechanism that simulates real-world unpredictability through missing, failing, or distracting tool functions, forcing agents to detect disrupted paths and adapt at runtime. Experiments on ten leading LLMs show that massive-tool planning remains challenging: while GPT-5.4 achieves 51.90% accuracy in block-free settings, it collapses to 11.36% under the most severe blocking condition. Further analysis shows that agents are especially vulnerable when failures lack explicit error signals or when recovery requires longer alternative tool-use paths. These results establish PlanBench-XL as a testbed for diagnosing agentic planning failures and highlight the need for robust adaptive planning in long-horizon tasks with large, imperfect tool environments.

Problem

• long-horizon planning
• tool-use agents
• large-scale tool ecosystems
• retrieval-limited visibility
• adaptive planning

Innovation

• long-horizon planning
• tool-use agents
• retrieval-limited visibility
• blocking mechanism
• adaptive planning