Tree Search for Language Model Agents

2024-07-01

arXiv.org

AI Summary

To address the limited multi-step decision-making capability of language model agents in real-world web environments—characterized by weak planning and insufficient utilization of environmental feedback—this paper proposes Environment-aware Best-First Search (Env-BFS), a reasoning-time, environment-integrated tree search algorithm. Env-BFS explicitly models the action space, enables interactive environment exploration and backtracking, and achieves computationally scalable planning via dynamic LM API scheduling. As the first tree search framework validated on realistic web benchmarks (VisualWebArena and WebArena), it features a plug-and-play design compatible with any state-of-the-art agent. Experiments demonstrate significant improvements: success rates increase by 39.7% to 26.4% on VisualWebArena and by 28.0% to 19.2% on WebArena—establishing new SOTA performance. The code and models are publicly released.

Technology Category

• Search and Optimization: Learning to Search
• Multiagent Systems: Multiagent Planning
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Autonomous agents powered by language models (LMs) have demonstrated promise in their ability to perform decision-making tasks such as web automation. However, a key limitation remains: LMs, primarily optimized for natural language understanding and generation, struggle with multi-step reasoning, planning, and using environmental feedback when attempting to solve realistic computer tasks. Towards addressing this, we propose an inference-time search algorithm for LM agents to explicitly perform exploration and multi-step planning in interactive web environments. Our approach is a form of best-first tree search that operates within the actual environment space, and is complementary with most existing state-of-the-art agents. It is the first tree search algorithm for LM agents that shows effectiveness on realistic web tasks. On the challenging VisualWebArena benchmark, applying our search algorithm on top of a GPT-4o agent yields a 39.7% relative increase in success rate compared to the same baseline without search, setting a state-of-the-art success rate of 26.4%. On WebArena, search also yields a 28.0% relative improvement over a baseline agent, setting a competitive success rate of 19.2%. Our experiments highlight the effectiveness of search for web agents, and we demonstrate that performance scales with increased test-time compute. We conduct a thorough analysis of our results to highlight improvements from search, limitations, and promising directions for future work. Our code and models are publicly released at https://jykoh.com/search-agents.

Problem

• LM agents struggle with multi-step reasoning in interactive environments
• Existing language models lack planning capabilities for realistic computer tasks
• Agents need better environmental feedback utilization for web automation

Innovation

• Best-first tree search in environment space
• Exploration and multi-step planning for agents
• Inference-time algorithm for web automation tasks