BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent

2025-08-08

AI Summary

Existing deep-reasoning agent benchmarks (e.g., BrowseComp) suffer from fairness and transparency deficiencies: they rely on opaque, dynamic search APIs—undermining reproducibility—and lack controlled document corpora, hindering disentanglement of retriever versus LLM contributions. To address these issues, we propose BrowseComp-Plus—the first open-source evaluation benchmark built upon a fixed, manually curated document corpus. It integrates reproducible retrievers including BM25 and Qwen3-Embedding-8B, and introduces human-verified supporting documents alongside challenging negative samples to enable precise decoupling of retrieval and reasoning capabilities. Experiments demonstrate that BrowseComp-Plus effectively discriminates model performance: GPT-5 augmented with Qwen3-Embedding-8B achieves 70.1% accuracy while substantially reducing search API calls, outperforming leading open-source alternatives.

Technology Category

• Search and Optimization: Learning to Search
• Natural Language Processing: Question Answering
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Deep-Research agents, which integrate large language models (LLMs) with search tools, have shown success in improving the effectiveness of handling complex queries that require iterative search planning and reasoning over search results. Evaluations on current benchmarks like BrowseComp relies on black-box live web search APIs, have notable limitations in (1) fairness: dynamic and opaque web APIs hinder fair comparisons and reproducibility of deep research methods; (2) transparency: lack of control over the document corpus makes it difficult to isolate retriever contributions. In other words, the current evaluations may compare a complete deep research system at a given time, but they do not foster well-controlled experiments to provide insights into the capability of underlying deep research LLMs. To address these challenges, we introduce BrowseComp-Plus, a benchmark derived from BrowseComp, employing a fixed, carefully curated corpus. Each query in BrowseComp-Plus includes human-verified supporting documents and mined challenging negatives, enabling controlled experimentation. The benchmark is shown to be effective in distinguishing the performance of deep research systems. For instance, the open-source model Search-R1, when paired with the BM25 retriever, achieves 3.86% accuracy, whereas the GPT-5 achieves 55.9%. Integrating the GPT-5 with the Qwen3-Embedding-8B retriever further enhances its accuracy to 70.1% with fewer search calls. This benchmark allows comprehensive evaluation and disentangled analysis of deep research agents and retrieval methods, fostering insights into retrieval effectiveness, citation accuracy, and context engineering in Deep-Research system.

Problem

• Fair comparison hindered by dynamic web APIs
• Lack of transparency in document corpus control
• Inability to isolate retriever contributions effectively

Innovation

• Fixed curated corpus for fair comparisons
• Human-verified documents and mined negatives
• Disentangled analysis of retrieval methods