Exploration and Exploitation Errors Are Measurable for Language Model Agents

2026-04-14

AI Summary

Current language model agents lack systematic means to distinguish and quantify exploration versus exploitation errors when internal policies are inaccessible. This work proposes a policy-agnostic evaluation framework that leverages embodied AI principles to construct a controllable, partially observable 2D grid environment. By integrating programmable map generation with task-agnostic directed acyclic graphs (DAGs) of unknown tasks, the framework dynamically modulates the difficulty of exploration or exploitation and defines quantifiable metrics for both error types based solely on observable behavior. This approach enables, for the first time, a decoupled assessment of language models’ exploration–exploitation behaviors, revealing significant yet divergent failure modes across state-of-the-art models. Notably, reasoning-oriented models exhibit superior performance, and their capabilities can be effectively enhanced through lightweight reasoning guidance.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Multiagent Systems: Adversarial Agents

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Language Model (LM) agents are increasingly used in complex open-ended decision-making tasks, from AI coding to physical AI. A core requirement in these settings is the ability to both explore the problem space and exploit acquired knowledge effectively. However, systematically distinguishing and quantifying exploration and exploitation from observed actions without access to the agent's internal policy remains challenging. To address this, we design controllable environments inspired by practical embodied AI scenarios. Each environment consists of a partially observable 2D grid map and an unknown task Directed Acyclic Graph (DAG). The map generation can be programmatically adjusted to emphasize exploration or exploitation difficulty. To enable policy-agnostic evaluation, we design a metric to quantify exploration and exploitation errors from agent's actions. We evaluate a variety of frontier LM agents and find that even state-of-the-art models struggle on our task, with different models exhibiting distinct failure modes. We further observe that reasoning models solve the task more effectively and show both exploration and exploitation can be significantly improved through minimal harness engineering. We release our code \href{https://github.com/jjj-madison/measurable-explore-exploit}{here}.

Problem

• Exploration
• Exploitation
• Language Model Agents
• Error Quantification
• Policy-Agnostic Evaluation

Innovation

• exploration-exploitation tradeoff
• language model agents
• policy-agnostic evaluation
• embodied AI
• measurable error metrics