Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning

2026-02-10

AI Summary

Current reinforcement learning agents are limited by the scarcity of diverse and reliable interactive environments. This work proposes the first large-scale, code-driven, database-backed framework for synthetic environment generation, automatically constructing 1,000 executable environments representing everyday scenarios, each integrating an average of 35 tools. By leveraging a database to enforce consistent and reliable state transitions, the framework enables efficient training of multi-turn tool-using agents with strong generalization capabilities—entirely without reliance on real-world or task-specific environments. Experimental results demonstrate that agents trained exclusively in these synthetic environments achieve strong out-of-distribution generalization across three benchmarks, validating the effectiveness and scalability of the proposed approach.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Multiagent Systems: Adversarial Agents
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent advances in large language model (LLM) have empowered autonomous agents to perform complex tasks that require multi-turn interactions with tools and environments. However, scaling such agent training is limited by the lack of diverse and reliable environments. In this paper, we propose Agent World Model (AWM), a fully synthetic environment generation pipeline. Using this pipeline, we scale to 1,000 environments covering everyday scenarios, in which agents can interact with rich toolsets (35 tools per environment on average) and obtain high-quality observations. Notably, these environments are code-driven and backed by databases, providing more reliable and consistent state transitions than environments simulated by LLMs. Moreover, they enable more efficient agent interaction compared with collecting trajectories from realistic environments. To demonstrate the effectiveness of this resource, we perform large-scale reinforcement learning for multi-turn tool-use agents. Thanks to the fully executable environments and accessible database states, we can also design reliable reward functions. Experiments on three benchmarks show that training exclusively in synthetic environments, rather than benchmark-specific ones, yields strong out-of-distribution generalization. The code is available at https://github.com/Snowflake-Labs/agent-world-model.

Problem

• synthetic environments
• agentic reinforcement learning
• environment scalability
• tool-use agents
• state consistency

Innovation

• synthetic environments
• agent world model
• agentic reinforcement learning
• code-driven simulation
• out-of-distribution generalization