Prompt-Driven Exploration

2026-07-09

AI Summary

This work addresses the challenge of inefficient exploration in traditional reinforcement learning, which relies on random action-space exploration and struggles to achieve effective global policy improvement in sparse or zero-reward environments. The authors propose a novel natural language–guided exploration framework that introduces posterior sampling exploration into the prompt space for the first time. By leveraging a vision-language model (VLM) to analyze execution videos of agent trajectories, the method diagnoses failure causes and automatically rewrites task prompts, thereby guiding a large language model (LLM) and a vision-language-action (VLA) model to generate more exploratory behaviors. This approach significantly enhances sample efficiency without requiring any initial reward signals and successfully learns effective policies across both manipulation and reasoning tasks.

Abstract

Exploration is essential to RL since a policy cannot improve by repeatedly sampling the behaviors it already prefers. Standard methods inject stochasticity in the action space, but such jitter only yields rollouts close to the original. Escaping a weak policy often requires global perturbations that action noise cannot produce. Large language models (LLMs) and vision-language-action (VLA) models offer a pathway: they condition the policy on a natural language prompt, and since the rollout follows from it, modifying the prompt induces global changes. The challenge is finding prompts that induce useful global changes. With a weak policy that rarely succeeds, reward is too sparse to select on. Our idea is to refine prompts from the rollouts themselves: a vision-language model (VLM) reasons over the rollout video, diagnoses how the policy responded, and rewrites the prompt to elicit better behavior next time. This procedure realizes posterior sampling, a classical RL exploration framework, at the level of prompts: the VLM maintains an implicit distribution over useful prompts and updates it from observed rollouts. We call this strategy Prompt-Driven Exploration (PDE). Across manipulation and reasoning tasks, PDE enables RL to learn successful policies even from zero-reward starts, and improves sample efficiency more broadly. Our website is available at https://xinyunsunshine.github.io/prompt-rl.

Problem

• reinforcement learning
• exploration
• sparse reward
• prompt engineering
• language-guided control

Innovation

• Prompt-Driven Exploration
• Vision-Language Model
• Posterior Sampling
• Reinforcement Learning
• Natural Language Prompt