AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning

2025-09-10

AI Summary

Large language models (LLMs) struggle to efficiently learn complex, long-horizon decision-making tasks from scratch. Method: This paper proposes Inter-RL—a unified, interactive multi-round reinforcement learning framework that eliminates reliance on supervised fine-tuning. It employs a modular decoupled architecture and the ScalingInter-RL training paradigm to dynamically balance exploration and exploitation, enhancing behavioral diversity and training stability; additionally, a progressive step-constraint strategy enables end-to-end optimization for extended decision horizons. Contribution/Results: Evaluated across 27 real-world, cross-domain tasks, LLM agents trained with Inter-RL achieve performance on par with or exceeding that of commercial models. All code and datasets are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Multiagent Learning
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Developing autonomous LLM agents capable of making a series of intelligent decisions to solve complex, real-world tasks is a fast-evolving frontier. Like human cognitive development, agents are expected to acquire knowledge and skills through exploration and interaction with the environment. Despite advances, the community still lacks a unified, interactive reinforcement learning (RL) framework that can effectively train such agents from scratch -- without relying on supervised fine-tuning (SFT) -- across diverse and realistic environments. To bridge this gap, we introduce AgentGym-RL, a new framework to train LLM agents for multi-turn interactive decision-making through RL. The framework features a modular and decoupled architecture, ensuring high flexibility and extensibility. It encompasses a wide variety of real-world scenarios, and supports mainstream RL algorithms. Furthermore, we propose ScalingInter-RL, a training approach designed for exploration-exploitation balance and stable RL optimization. In early stages, it emphasizes exploitation by restricting the number of interactions, and gradually shifts towards exploration with larger horizons to encourage diverse problem-solving strategies. In this way, the agent develops more diverse behaviors and is less prone to collapse under long horizons. We perform extensive experiments to validate the stability and effectiveness of both the AgentGym-RL framework and the ScalingInter-RL approach. Our agents match or surpass commercial models on 27 tasks across diverse environments. We offer key insights and will open-source the complete AgentGym-RL framework -- including code and datasets -- to empower the research community in developing the next generation of intelligent agents.

Problem

• Training LLM agents for long-horizon decision making without supervised fine-tuning
• Lacking unified interactive RL framework for diverse real-world environments
• Balancing exploration-exploitation in multi-turn reinforcement learning training

Innovation

• Multi-turn reinforcement learning framework for LLM agents
• Modular and decoupled architecture for flexibility
• ScalingInter-RL approach balances exploration and exploitation