Toward Efficient Exploration by Large Language Model Agents

2025-04-29

AI Summary

Large language model (LLM) agents suffer from low exploration efficiency and poor data utilization in natural language reinforcement learning (NL-RL). Method: This paper proposes the first framework that explicitly implements posterior sampling reinforcement learning (PSRL) entirely via LLMs—without fine-tuning or in-context learning. It delegates the full Bayesian update and action-sampling process of PSRL to the LLM using natural language, enabling interpretable, statistically robust zero-shot exploration in natural language state-action spaces. Contribution/Results: The core innovation is a theory-driven algorithm orchestration mechanism grounded in natural language semantics, requiring no model adaptation. Experiments on multi-turn language tasks demonstrate a 2.3× improvement in sample efficiency over state-of-the-art LLM-RL agents, validating both the effectiveness and generalizability of explicitly deploying principled RL algorithms within language agents.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Multiagent Learning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

A burgeoning area within reinforcement learning (RL) is the design of sequential decision-making agents centered around large language models (LLMs). While autonomous decision-making agents powered by modern LLMs could facilitate numerous real-world applications, such successes demand agents that are capable of data-efficient RL. One key obstacle to achieving data efficiency in RL is exploration, a challenge that we demonstrate many recent proposals for LLM agent designs struggle to contend with. Meanwhile, classic algorithms from the RL literature known to gracefully address exploration require technical machinery that can be challenging to operationalize in purely natural language settings. In this work, rather than relying on finetuning or in-context learning to coax LLMs into implicitly imitating a RL algorithm, we illustrate how LLMs can be used to explicitly implement an existing RL algorithm (Posterior Sampling for Reinforcement Learning) whose capacity for statistically-efficient exploration is already well-studied. We offer empirical results demonstrating how our LLM-based implementation of a known, data-efficient RL algorithm can be considerably more effective in natural language tasks that demand prudent exploration.

Problem

• Improving data-efficient RL exploration for LLM agents
• Addressing exploration challenges in autonomous decision-making agents
• Implementing Posterior Sampling for RL with LLMs

Innovation

• LLM-based implementation of Posterior Sampling
• Explicit use of existing RL algorithm
• Enhanced exploration in natural language tasks