LLM-Guided Reinforcement Learning: Addressing Training Bottlenecks through Policy Modulation

2025-05-27

AI Summary

Reinforcement learning (RL) often converges to suboptimal policies in complex tasks, failing to maximize long-term cumulative reward. Existing policy optimization approaches either incur high computational costs through automated strategy search or suffer from poor scalability due to reliance on manual human feedback. To address these limitations, we propose an LLM-guided zero-shot policy modulation framework that requires neither additional model training nor human intervention. Leveraging large language models (LLMs) via prompt engineering, our method automatically identifies critical states, generates actionable recommendations, and implicitly assigns rewards—all without modifying the underlying RL agent. It seamlessly integrates with standard RL algorithms such as PPO and SAC. To the best of our knowledge, this is the first work to enable end-to-end, implicit LLM guidance over the entire policy optimization pipeline. Empirical evaluation across multiple benchmark tasks demonstrates substantial improvements over state-of-the-art methods, including accelerated convergence and superior final performance—establishing a scalable, low-cost paradigm for overcoming fundamental RL training bottlenecks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

While reinforcement learning (RL) has achieved notable success in various domains, training effective policies for complex tasks remains challenging. Agents often converge to local optima and fail to maximize long-term rewards. Existing approaches to mitigate training bottlenecks typically fall into two categories: (i) Automated policy refinement, which identifies critical states from past trajectories to guide policy updates, but suffers from costly and uncertain model training; and (ii) Human-in-the-loop refinement, where human feedback is used to correct agent behavior, but this does not scale well to environments with large or continuous action spaces. In this work, we design a large language model-guided policy modulation framework that leverages LLMs to improve RL training without additional model training or human intervention. We first prompt an LLM to identify critical states from a sub-optimal agent's trajectories. Based on these states, the LLM then provides action suggestions and assigns implicit rewards to guide policy refinement. Experiments across standard RL benchmarks demonstrate that our method outperforms state-of-the-art baselines, highlighting the effectiveness of LLM-based explanations in addressing RL training bottlenecks.

Problem

• Overcoming local optima in RL training
• Reducing reliance on costly human feedback
• Improving policy refinement without additional training

Innovation

• LLM identifies critical states from trajectories
• LLM provides action suggestions for policy refinement
• LLM assigns implicit rewards to guide training