WANDR: A Benchmark for Wide and Deep Research

2026-08-13

AI Summary

This study addresses the absence of effective evaluation benchmarks for AI agents in broad-and-deep data collection tasks by constructing a specialized benchmark comprising 500 tasks. We introduce a Qualification Key Hierarchical Representation and a dynamic evaluation mechanism grounded in real-time evidence verification, integrated with a semi-automated generation pipeline to precisely quantify agents' breadth and depth retrieval capabilities. Experimental results demonstrate that the current state-of-the-art system achieves a Soft F1 score of only 0.363, revealing significant bottlenecks in large-scale hierarchical retrieval. These findings establish a critical benchmark and provide methodological foundations for advancing research in complex information-seeking agents.

Abstract

WANDR (Wide ANd Deep Research) is a benchmark of 500 realistic, challenging data-collection tasks for research agents. Each task requires a system to discover a large set of entities that satisfy specified criteria (breadth), investigate each entity through multiple coordinated web searches (depth), and return independently verifiable records with supporting sources and excerpts. Tasks are represented as qualification key hierarchies that specify the entities, relationships, evidence, and required count at each level; a hierarchy with n companies, m employees per company, and k sources per employee requires n x m x k records. This structure supports diverse workflows such as market mapping, due diligence, literature review, product comparison, and talent sourcing, with targets ranging from dozens to thousands of records. WANDR replaces static gold answer sets with task-specific judges that refetch cited pages and verify each record against its evidence, allowing evaluation of current and changing facts. Record verdicts are aggregated into soft and hard precision, recall, and F1 scores that distinguish factual quality, coverage, and hierarchical completeness. The tasks are derived from de-identified product-usage logs and produced through a semi-automated pipeline with automated checks, empirical audits, and human review where needed. We evaluate six production research systems and find that the benchmark is far from saturated: at high effort, the strongest system reaches only 0.363 soft F1 and 0.133 hard F1. Performance degrades as target volume and hierarchy depth increase, with incomplete discovery, missing enrichment, and incomplete evidence construction remaining major bottlenecks. The benchmark and evaluation harness are available at https://github.com/perplexityai/wandr.

Problem

• Research Agents
• Benchmark
• Data Collection
• Evaluation
• Wide and Deep Research

Innovation

• Wide and Deep Research Benchmark
• Qualification Key Hierarchy
• Dynamic Task-Specific Judges
• Hierarchical Evaluation Metrics
• Semi-Automated Task Generation