AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments

2026-07-06

AI Summary

Current evaluations of large language model agents predominantly rely on idealized environments that fail to capture real-world challenges such as noise, incomplete information, and tool exploration. To address this gap, this work proposes AgentGym2—a high-fidelity, end-to-end evaluation framework that systematically assesses agent capabilities under de-idealized conditions. For the first time, it evaluates comprehensive competencies including dynamic tool discovery, cross-task tool composition, robustness to ambiguous or noisy inputs, and proactive exploration behavior. The study benchmarks 15 prominent open- and closed-source models, including Gemini and GPT-5, revealing significant performance deficiencies in realistic tasks and highlighting a critical gap between current agent capabilities and practical deployment requirements.

Abstract

Language agents, i.e., LLM agents, progress rapidly and are increasingly deployed in production environments. This trend underscores the urgent need for rigorous and realistic evaluations. However, most existing benchmarks evaluate agents in simplified, idealized settings. They typically rely on pre-packaged tool interfaces, overlook critical steps, and assume inputs are clean and fully specified. Consequently, they understate the difficulty of real deployments, where uncertainty and noise are ubiquitous and agents must proactively explore the environment to uncover new tools. To bridge this gap, we present AgentGym2, a new evaluation framework with task instances grounded in real-world end-to-end working demands. Beyond reasoning and planning, it measures agents' ability to execute end-to-end procedures, discover tools via exploration, compose tools for unseen tasks, and remain robust to noisy and underspecified information. Experiments on 15 proprietary and open-source models show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2, revealing a substantial gap between the capability of current agents and the demands of real-world applications.

Problem

• LLM agents
• real-world environments
• evaluation benchmark
• tool discovery
• noisy inputs

Innovation

• real-world evaluation
• tool discovery
• end-to-end execution
• LLM agents
• de-idealized environments