Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipe

2026-03-23

AI Summary

Existing approaches struggle to effectively scale reinforcement learning (RL) for training large language model (LLM) agents with long-horizon planning capabilities in complex, multi-turn environments. This work proposes TravelPlanner, a benchmark platform that systematically explores the RL design space across five critical dimensions: reward design, model scale, data composition, algorithm selection, and environment stability, yielding a reproducible training recipe. Experiments reveal that only around 1K difficulty-balanced samples are sufficient to achieve strong in-domain and out-of-domain performance; model scale exhibits strong interdependence with reward and algorithm choices; and environment instability significantly exacerbates policy degradation. Models trained using this recipe attain state-of-the-art performance on TravelPlanner, substantially outperforming mainstream LLMs.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Multiagent Planning

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions

Abstract

Reinforcement Learning (RL) is essential for evolving Large Language Models (LLMs) into autonomous agents capable of long-horizon planning, yet a practical recipe for scaling RL in complex, multi-turn environments remains elusive. This paper presents a systematic empirical study using TravelPlanner, a challenging testbed requiring tool orchestration to satisfy multifaceted constraints. We decompose the agentic RL design space along 5 axes: reward shaping, model scaling, data composition, algorithm selection, and environmental stability. Our controlled experiments yield 7 key takeaways, e.g., (1) reward and algorithm choices are scale-dependent as smaller models benefit from staged rewards and enhanced exploration, whereas larger models converge efficiently with simpler dense rewards, (2) ~ 1K training samples with a balanced difficulty mixture mark a sweet spot for both in-domain and out-of-domain performance, and (3) environmental stability is critical to prevent policy degradation. Based on our distilled recipe, our RL-trained models achieve state-of-the-art performance on TravelPlanner, significantly outperforming leading LLMs.

Problem

• Reinforcement Learning
• Long-Horizon Planning
• Tool-Using Agents
• Large Language Models
• Multi-turn Environments

Innovation

• Reinforcement Learning
• Long-Horizon Planning
• Tool-Using Agents
• Reward Shaping
• Environmental Stability