AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web

2026-04-12

AI Summary

This work addresses the paradigm shift from centralized retrieval to multi-agent collaboration by introducing AgentWebBench, the first benchmark for evaluating coordinated multi-agent systems in the Agentic Web. It encompasses four task categories: web search, recommendation, question answering, and in-depth research. The authors construct an interaction framework between user and content agents powered by large language models (LLMs), integrating diverse coordination strategies, test-time scaling, and structured API interfaces. Systematic evaluations across seven state-of-the-art LLMs and three coordination strategies assess information synthesis capabilities in decentralized settings. Results demonstrate that multi-agent approaches can outperform centralized retrieval in question answering, that larger model scales help narrow performance gaps, and that planning, synthesis, retrieval, and evidence quality represent critical directions for future improvement.

Technology Category

• Multiagent Systems: Coordination and Collaboration
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Agentic Web is an emerging paradigm where autonomous agents help users use online information. As the paradigm develops, content providers are also deploying agents to manage their data and serve it through controlled interfaces. This shift moves information access from centralized retrieval to decentralized coordination. To study this setting, we introduce AgentWebBench, a benchmark that evaluates how well a user agent synthesizes answers by interacting with website-specific content agents. We evaluate four tasks that cover common web information needs, spanning ranked retrieval (web search, web recommendation) and open-ended synthesis (question answering, deep research). Across seven advanced LLMs and three coordination strategies, multi-agent coordination generally lags behind centralized retrieval as expected, because user agent cannot directly access the corpus, but the gap shrinks with model scale and can even outperform centralized retrieval on question answering. This benchmark also enables us to study properties of the emerging paradigm of the digital world. We find that decentralized access concentrates traffic toward a small set of websites, test time scaling improves both interaction reliability and task performance, and strong results require sufficient interactions guided by careful planning. Finally, our failure analysis suggests that user agents need better planning and answer synthesis, while content agents need more reliable retrieval and evidence quality. Code, data, and APIs are released on https://github.com/cxcscmu/AgentWebBench.

Problem

• Agentic Web
• multi-agent coordination
• decentralized information access
• agent benchmarking
• user-content agent interaction

Innovation

• multi-agent coordination
• Agentic Web
• decentralized information access
• agent benchmarking
• test-time scaling