WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation

2026-07-07

AI Summary

Existing benchmarks for web-based agents suffer from limited scale, narrow domain coverage, coarse-grained evaluation metrics, and insufficient consideration of real-world deployment requirements, hindering comprehensive assessment of cross-domain generalization and practical utility. To address these limitations, this work introduces WebRetriever—a large-scale, multi-domain benchmark encompassing 800 websites and 1,550 tasks—alongside NavEval, a novel evaluation framework that integrates interaction trajectories, query semantics, and operational context to overcome the constraints of conventional approaches relying solely on screenshots or navigation success rates. Through three complementary protocols—navigation capability, knowledge-augmented interaction, and end-to-end task completion—the framework enables fine-grained, human-aligned evaluation. Experiments reveal significant performance disparities across protocols, demonstrating that navigation success alone inadequately reflects real-world effectiveness; WebRetriever shows strong alignment with human judgments across multiple datasets, offering a robust foundation for agent development and evaluation.

Abstract

As web agents increasingly demonstrate capabilities in automated task execution, the development of robust evaluation frameworks for assessing their navigation and task completion performance has emerged as a critical research priority. However, existing benchmarks exhibit fundamental limitations. First, they suffer from insufficient scale and limited domain diversity, constraining comprehensive evaluation of cross-domain generalization. Second, prevailing LLM-as-Judge evaluation methodologies inadequately capture fine-grained interaction semantics, particularly regarding precise query formulation and filtering operations. Third, current benchmarks predominantly emphasize navigation success metrics while neglecting critical requirements for real-world deployment scenarios. To address these limitations, we introduce WebRetriever, a large-scale benchmark encompassing 800 websites and 1,550 tasks across diverse domains, including consumer, professional, and enterprise sectors, with comprehensive coverage of user intent patterns. We propose NavEval (Navigation Evaluation), a novel LLM-as-Judge framework that leverages rich interaction context beyond visual screenshots, achieving state-of-the-art alignment with human judgment across multiple evaluation datasets. Furthermore, we establish three complementary evaluation protocols that collectively provide holistic assessment of web agent capabilities: navigation proficiency, knowledge-assisted interaction, and end-to-end task completion with information extraction. Extensive experimental analysis reveals substantial performance disparities across evaluation protocols, demonstrating that navigation success alone is an insufficient predictor of real-world application effectiveness. WebRetriever delivers fine-grained diagnostic insights into agent capabilities and establishes a rigorous foundation for advancing web agent research and development.

Problem

• web agent evaluation
• benchmark limitations
• cross-domain generalization
• interaction semantics
• real-world deployment

Innovation

• WebRetriever
• NavEval
• LLM-as-Judge
• web agent evaluation
• interaction semantics