LaGO: Latent Action Guidance for Online Reinforcement Learning

2026-06-23

AI Summary

This work addresses the unreliability of directly deploying large language models (LLMs) as reinforcement learning controllers, which often stems from imprecise action generation. To overcome this limitation, the authors propose LaGO, a novel framework that leverages a pretrained LLM as an implicit action prior to softly guide policy optimization online, rather than using it as the controller itself. LaGO integrates the LLM’s prior knowledge with the Proximal Policy Optimization (PPO) algorithm through a latent-space action guidance mechanism, enabling efficient policy learning. Evaluated on the CLEVR-Robot and Meta-World benchmarks, LaGO substantially improves performance, increasing average success rates from 15.1% to 27.2% and from 2.7% to 15.2%, respectively, thereby demonstrating its effectiveness and generalizability.

Abstract

Large language models (LLMs) have shown strong potential for planning and sequential decision-making, but prior work often relies on using them as direct controllers, which requires precise action generation and can be unreliable in practice. This paper proposes Latent Action Guidance for Online Reinforcement Learning (LaGO), a framework that uses a pretrained LLM as a latent action prior to softly guide online policy optimization, rather than treating the LLM as an explicit planner or controller. Experiments on both a discrete-control benchmark, CLEVR-Robot, and a continuous-control benchmark, Meta-World, demonstrate that LaGO consistently improves both reward and success rate over Vanilla PPO. In particular, LaGO increases the average success rate from 15.1% to 27.2% on CLEVR-Robot and from 2.7% to 15.2% on Meta-World. Our analysis further shows that stronger pretrained LLMs provide more effective guidance, suggesting that LLM knowledge can improve planning and online decision-making.

Problem

• Large Language Models
• Online Reinforcement Learning
• Latent Action Guidance
• Sequential Decision-Making
• Policy Optimization

Innovation

• Latent Action Guidance
• Online Reinforcement Learning
• Large Language Models
• Policy Optimization
• Pretrained Priors