Reinforcement Learning for Long-Horizon Interactive LLM Agents

2025-02-03

AI Summary

Existing LLM-based agents exhibit low task success rates (<50%) and suffer from hallucination and error propagation in multi-application, stateful digital environments (e.g., AppWorld). To address this, this work introduces reinforcement learning (RL) for the first time into training long-horizon, API-driven agents across multiple domains. We propose LOOP—a lightweight, value-network-free variant of PPO that operates with a single LLM instance and is optimized for memory and data efficiency—within an end-to-end RL framework grounded in partially observable Markov decision process (POMDP) modeling. LOOP integrates API documentation awareness and robust error recovery mechanisms. We conduct full-model RL training on a 32B-parameter LLM and achieve a 9-percentage-point absolute improvement (15% relative gain) in task success rate over OpenAI o1 on the AppWorld benchmark, significantly enhancing long-horizon reasoning robustness and interaction quality.

Technology Category

• Machine Learning: Reinforcement Learning
• Humans and AI: Human-in-the-loop Machine Learning
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Interactive digital agents (IDAs) leverage APIs of stateful digital environments to perform tasks in response to user requests. While IDAs powered by instruction-tuned large language models (LLMs) can react to feedback from interface invocations in multi-step exchanges, they have not been trained in their respective digital environments. Prior methods accomplish less than half of tasks in sophisticated benchmarks such as AppWorld. We present a reinforcement learning (RL) approach that trains IDAs directly in their target environments. We formalize this training as a partially observable Markov decision process and derive LOOP, a data- and memory-efficient variant of proximal policy optimization. LOOP uses no value network and maintains exactly one copy of the underlying LLM in memory, making its implementation straightforward and as memory-efficient as fine-tuning a single LLM. A 32-billion-parameter agent trained with LOOP in the AppWorld environment outperforms the much larger OpenAI o1 agent by 9 percentage points (15% relative). To our knowledge, this is the first reported application of RL to IDAs that interact with a stateful, multi-domain, multi-app environment via direct API calls. Our analysis sheds light on the effectiveness of RL in this area, showing that the agent learns to consult the API documentation, avoid unwarranted assumptions, minimize confabulation, and recover from setbacks.

Problem

• Reinforcement Learning
• Large Model Agents
• Multi-domain Applications

Innovation

• Reinforcement Learning
• M-PPO Efficient Learning Method
• Interactive Digital Assistants