AutoForge: Automated Environment Synthesis for Agentic Reinforcement Learning

2025-12-28

AI Summary

Current reinforcement learning (RL) for language agents is hindered by two key bottlenecks: (i) synthetic simulation environments are only semi-automated, lack task difficulty, and suffer from unstable simulated users; and (ii) high environmental heterogeneity impedes generalization and robustness. Method: We propose the first fully automated, scalable, and high-difficulty simulation environment synthesis framework. It introduces a unified automation pipeline integrating large language model (LLM)-driven task modeling and environment generation. We further design an environment-level RL algorithm featuring environment-level advantage estimation, uncertainty-aware user behavior modeling, and robust policy optimization. Contribution/Results: Our framework achieves significant performance gains across multiple benchmarks—including tau-bench, tau2-Bench, and VitaBench—demonstrating strong training stability and cross-domain generalization capability.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Conducting reinforcement learning (RL) in simulated environments offers a cost-effective and highly scalable way to enhance language-based agents. However, previous work has been limited to semi-automated environment synthesis or tasks lacking sufficient difficulty, offering little breadth or depth. In addition, the instability of simulated users integrated into these environments, along with the heterogeneity across simulated environments, poses further challenges for agentic RL. In this work, we propose: (1) a unified pipeline for automated and scalable synthesis of simulated environments associated with high-difficulty but easily verifiable tasks; and (2) an environment level RL algorithm that not only effectively mitigates user instability but also performs advantage estimation at the environment level, thereby improving training efficiency and stability. Comprehensive evaluations on agentic benchmarks, including tau-bench, tau2-Bench, and VitaBench, validate the effectiveness of our proposed method. Further in-depth analyses underscore its out-of-domain generalization.

Problem

• Automated synthesis of high-difficulty simulated environments for agentic reinforcement learning
• Mitigating user instability and heterogeneity across simulated environments in RL
• Improving training efficiency and stability through environment-level advantage estimation

Innovation

• Automated pipeline synthesizes high-difficulty verifiable tasks
• Environment-level RL algorithm mitigates user instability
• Advantage estimation at environment level improves training efficiency