DR$^{3}$-Eval: Towards Realistic and Reproducible Deep Research Evaluation

2026-04-16

AI Summary

This work addresses the challenge of evaluating deep research agents in complex, long-horizon scientific tasks, where dynamic environments and ambiguous objectives hinder reliable assessment. To this end, we introduce DR³-Eval, the first evaluation benchmark that balances realism with reproducibility by constructing a static research sandbox corpus derived from real user materials, enabling multimodal, multi-document report generation. The benchmark features a controlled environment incorporating supporting documents, distractors, and noise, alongside a multidimensional automatic evaluation framework aligned with human judgment—assessing information recall, factual accuracy, citation coverage, instruction adherence, and depth of analysis. Experimental results reveal significant shortcomings in current approaches, particularly in retrieval robustness and hallucination control. The code and data are publicly released.

Technology Category

• Search and Optimization: Evaluation and Analysis
• Machine Learning: Evaluation and Analysis
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation

Abstract

Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation, yet their evaluation remains challenging due to dynamic web environments and ambiguous task definitions. We propose DR$^{3}$-Eval, a realistic and reproducible benchmark for evaluating deep research agents on multimodal, multi-file report generation. DR$^{3}$-Eval is constructed from authentic user-provided materials and paired with a per-task static research sandbox corpus that simulates open-web complexity while remaining fully verifiable, containing supportive documents, distractors, and noise. Moreover, we introduce a multi-dimensional evaluation framework measuring Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality, and validate its alignment with human judgments. Experiments with our developed multi-agent system DR$^{3}$-Agent based on multiple state-of-the-art language models demonstrate that DR$^{3}$-Eval is highly challenging and reveals critical failure modes in retrieval robustness and hallucination control. Our code and data are publicly available.

Problem

• Deep Research Agents
• Evaluation Benchmark
• Multimodal Report Generation
• Reproducibility
• Realistic Assessment

Innovation

• Deep Research Agents
• Realistic Evaluation
• Reproducible Benchmark
• Multimodal Report Generation
• Multi-dimensional Evaluation