GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows

2026-04-17

AI Summary

Existing tool-use benchmarks often lack realism, making it difficult to evaluate the capabilities of general-purpose agents in complex, real-world productivity workflows. This work proposes GTA-2, a hierarchical benchmark comprising atomic-level tool use (GTA-Atomic) and open-ended, long-horizon workflows (GTA-Workflow), grounded in authentic user queries, deployed tools, and multimodal context. GTA-2 introduces a novel recursive checkpointing mechanism to jointly assess both model capabilities and execution frameworks. Experimental results reveal that state-of-the-art models achieve less than 50% success on atomic tasks and only 14.39% on workflow tasks. However, incorporating checkpoint-based feedback and advanced execution frameworks—such as Manus and OpenClaw—substantially improves performance, underscoring the critical role of execution architecture in task success.

Technology Category

• Multiagent Systems: Teamwork
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The development of general-purpose agents requires a shift from executing simple instructions to completing complex, real-world productivity workflows. However, current tool-use benchmarks remain misaligned with real-world requirements, relying on AI-generated queries, dummy tools, and limited system-level coordination. To address this, we propose GTA-2, a hierarchical benchmark for General Tool Agents (GTA) spanning atomic tool use and open-ended workflows. Built on real-world authenticity, it leverages real user queries, deployed tools, and multimodal contexts. (i) GTA-Atomic, inherited from our prior GTA benchmark, evaluates short-horizon, closed-ended tool-use precision. (ii) GTA-Workflow introduces long-horizon, open-ended tasks for realistic end-to-end completion. To evaluate open-ended deliverables, we propose a recursive checkpoint-based evaluation mechanism that decomposes objectives into verifiable sub-goals, enabling unified evaluation of both model capabilities and agent execution frameworks (i.e., execution harnesses). Experiments reveal a pronounced capability cliff: while frontier models already struggle on atomic tasks (below 50%), they largely fail on workflows, with top models achieving only 14.39% success. Further analysis shows that checkpoint-guided feedback improves performance, while advanced frameworks such as Manus and OpenClaw substantially enhance workflow completion, highlighting the importance of execution harness design beyond the underlying model capacity. These findings provide guidance for developing reliable personal and professional assistants. Dataset and code will be available at https://github.com/open-compass/GTA.

Problem

• tool-use benchmark
• general-purpose agents
• real-world workflows
• open-ended tasks
• evaluation misalignment

Innovation

• General Tool Agents
• Tool-use Benchmarking
• Recursive Checkpoint Evaluation
• Open-ended Workflows
• Execution Harness