Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments

2026-02-12

AI Summary

Current evaluations of large language model (LLM) agents are largely confined to static or synchronous environments, failing to capture real-world challenges such as dynamic environmental evolution and asynchronous event occurrence. This work proposes Gaia2, a novel benchmark that enables the first systematic evaluation of LLM agents in dynamic, asynchronous scenarios within consumer-grade settings. Built upon the open-source Agents Research Environments (ARE) platform, Gaia2 introduces a write-action validator, asynchronous event simulation, and a multi-agent interaction framework, supporting fine-grained reinforcement learning training. Experimental results show that GPT-5 (high) achieves the highest pass@1 rate at 42% but underperforms on time-sensitive tasks, while Kimi-K2 emerges as the best open-source model with a 21% pass@1 rate, revealing fundamental trade-offs among reasoning capability, efficiency, and robustness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Adversarial Agents
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We introduce Gaia2, a benchmark for evaluating large language model agents in realistic, asynchronous environments. Unlike prior static or synchronous evaluations, Gaia2 introduces scenarios where environments evolve independently of agent actions, requiring agents to operate under temporal constraints, adapt to noisy and dynamic events, resolve ambiguity, and collaborate with other agents. Each scenario is paired with a write-action verifier, enabling fine-grained, action-level evaluation and making Gaia2 directly usable for reinforcement learning from verifiable rewards. Our evaluation of state-of-the-art proprietary and open-source models shows that no model dominates across capabilities: GPT-5 (high) reaches the strongest overall score of 42% pass@1 but fails on time-sensitive tasks, Claude-4 Sonnet trades accuracy and speed for cost, Kimi-K2 leads among open-source models with 21% pass@1. These results highlight fundamental trade-offs between reasoning, efficiency, robustness, and expose challenges in closing the"sim2real"gap. Gaia2 is built on a consumer environment with the open-source Agents Research Environments platform and designed to be easy to extend. By releasing Gaia2 alongside the foundational ARE framework, we aim to provide the community with a flexible infrastructure for developing, benchmarking, and training the next generation of practical agent systems.

Problem

• LLM agents
• dynamic environments
• asynchronous environments
• benchmarking
• sim2real gap

Innovation

• asynchronous environments
• dynamic benchmarking
• write-action verifier
• reinforcement learning from verifiable rewards
• sim2real gap