DORA Explorer: Improving the Exploration Ability of LLMs Without Training

2026-04-19

AI Summary

This work addresses the challenge of limited output diversity in large language models (LLMs) during sequential decision-making, which often leads to suboptimal policies or cyclic behavior. The authors propose a training-free exploration framework that generates diverse action candidates, scores them based on token log-probabilities, and selects actions via an adjustable exploration parameter, thereby achieving sequence-level diversity without any model training. By integrating action ranking with a controllable exploration mechanism, the method overcomes limitations inherent in conventional decoding and prompting strategies. Empirical results demonstrate its effectiveness: it matches the performance of Upper Confidence Bound (UCB) algorithms in multi-armed bandit tasks and substantially improves success rates in text-based adventure environments such as TextWorld—for instance, boosting the Qwen2.5-7B model’s success rate from 29.2% to 45.5%.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Sequential Decision Making
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Despite the rapid progress, LLMs for sequential decision-making (i.e., LLM agents) still struggle to produce diverse outputs. This leads to insufficient exploration, convergence to sub-optimal solutions, and becoming stuck in loops. Such limitations can be problematic in environments that require active exploration to gather information and make decisions. Sampling methods such as temperature scaling introduce token-level randomness but fail to produce enough diversity at the sequence level. We analyze LLM exploration in the classic Multi-Armed Bandit (MAB) setting and the Text Adventure Learning Environment Suite (TALES). We find that current decoding strategies and prompting methods like Chain-of-Thought and Tree-of-Thought are insufficient for robust exploration. To address this, we introduce DORA Explorer (Diversity-Oriented Ranking of Actions), a training-free framework for improving exploration in LLM agents. DORA generates diverse action candidates, scores them using token log-probabilities, and selects actions using a tunable exploration parameter. DORA achieves UCB-competitive performance on MAB and consistent gains across TALES, e.g., improving Qwen2.5-7B's performance from 29.2% to 45.5% in TextWorld. Our project is available at: https://dora-explore.github.io/.

Problem

• LLM agents
• exploration
• diversity
• sequential decision-making
• sub-optimal solutions

Innovation

• LLM agent
• exploration
• training-free
• diverse action generation
• sequence-level diversity