Clearing the Fog: Towards Installing and Refining Proactive Exploration Capabilities in LLM Agents

2026-08-14

AI Summary

This study addresses the limitations of insufficient active exploration and hindsight bias in LLM-based agents by proposing a novel framework that integrates exploratory data construction with contrastive signal-guided reinforcement learning. Specifically, the method synthesizes exploratory trajectories to mitigate bias and employs contrastive learning to distinguish effective exploratory behaviors, thereby optimizing decision-making policies. Experimental results validate the effectiveness of this approach, demonstrating significant improvements in the agents' active exploration capabilities. Beyond resolving existing bottlenecks, this research elucidates critical characteristics of active exploration, offering both a new technical pathway and theoretical foundation for enhancing the autonomy of LLM agents.

Abstract

We study proactive exploration in LLM agents, i.e., the ability to explore an environment to acquire information that improves future decision-making. In this regard, we first identify two fundamental bottlenecks that hinder this capability and then propose \ours, a novel method designed to instill and refine proactive exploration. Specifically, \ours\ consists of two components: (1) Exploratory Data Construction, which synthesizes exploration-rich trajectories to mitigate the hindsight bias of standard demonstrations; and (2) RL Optimization with Contrastive Signal Guidance, which leverages contrastive trajectory pairs to distinguish productive exploration from redundant wandering. Extensive experiments demonstrate the effectiveness of \ours\ and provide insights into the characteristics of proactive exploration. Our code is available at: https://github.com/GuanZhizhao/SAFARI.

Problem

• Proactive Exploration
• LLM Agents
• Hindsight Bias
• Information Acquisition

Innovation

• Proactive Exploration
• LLM Agents
• Exploratory Data Construction
• Contrastive RL Optimization
• Hindsight Bias Mitigation