Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge

2025-06-26

AI Summary

Existing benchmarks inadequately support open-ended, dynamic, and long-horizon agentic search due to restrictive assumptions—short search scopes, static answers, and no source attribution capability. This paper introduces Mind2Web 2, the first long-horizon benchmark tailored for large language model–driven autonomous web browsing and information synthesis, comprising 130 real-world tasks. We propose an innovative *Agent-as-a-Judge* evaluation framework: task-specific judge agents are automatically generated via hierarchical scoring rules to jointly assess answer correctness and source traceability. The benchmark integrates a realistic web interaction environment and enables cross-system comparative evaluation. Evaluated across nine state-of-the-art systems, OpenAI’s Deep Research achieves 50–70% of human performance while halving execution time. This work establishes the first automated, long-horizon, and attributable agentic search evaluation paradigm.

Technology Category

• Search and Optimization: Learning to Search
• Multiagent Systems: Adversarial Agents
• Data Mining & Knowledge Management: Knowledge Acquisition from the Web

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

Agentic search such as Deep Research systems, where large language models autonomously browse the web, synthesize information, and return comprehensive citation-backed answers, represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks and methodologies, which largely assume short search horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1,000 hours of human labor. To address the challenge of evaluating time-varying and complex answers, we propose a novel Agent-as-a-Judge framework. Our method constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution. We conduct a comprehensive evaluation of nine frontier agentic search systems and human performance, along with a detailed error analysis to draw insights for future development. The best-performing system, OpenAI Deep Research, can already achieve 50-70% of human performance while spending half the time, showing a great potential. Altogether, Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems.

Problem

• Evaluating complex agentic web search systems
• Assessing long-horizon dynamic information synthesis
• Automating correctness and source attribution judgment

Innovation

• Agent-as-a-Judge framework for evaluation
• Tree-structured rubric for answer assessment
• Real-time web browsing benchmark tasks