ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments

2026-06-18

AI Summary

Existing benchmarks struggle to systematically evaluate the performance of academic search agents in realistic, open-literature settings. This work proposes a large-scale academic search benchmark grounded in disciplinary taxonomies, introducing for the first time a taxonomy-guided query construction mechanism that supports four research intents: methodology-oriented, scenario-anchored, comparative analysis, and scope-controlled queries. The benchmark integrates a unified retrieval backend, ScholarBase, and a multi-metric evaluation framework—including Recall@100 and Recall@All. Experimental results demonstrate that agent-based approaches significantly outperform single-turn retrieval baselines; however, even the best-performing model achieves only 0.314 Recall@100, underscoring the task’s inherent difficulty. Comprehensive analyses of efficiency, robustness, and failure cases further provide rich diagnostic signals for future research.

Abstract

Academic paper search is a core step in scientific research, and LLM-based search agents are emerging as a promising paradigm for iterative, intent-driven literature exploration. However, existing benchmarks are insufficient for systematically evaluating agentic academic search under realistic open literature environments. We propose ScholarQuest, a large-scale, taxonomy-guided benchmark for agentic academic paper search. ScholarQuest is constructed from over 1,000 computer science topics and four representative research intents, including method-oriented, setting-anchored, comparison-based, and scope-controlled queries. It further provides scalable answer construction and a shared retrieval backend ScholarBase for reproducible evaluation. Benchmarking results show that agentic methods outperform single-shot retrieval baselines, yet the best-performing agent only achieves 0.314 Recall@100 and 0.355 Recall@All, indicating substantial room for improvement. In addition, analyses of search efficiency, intent-level robustness, and failure cases further highlight the benchmark's ability to provide multi-dimensional evaluation signals for academic paper search agents.

Problem

• academic paper search
• agentic search
• benchmark
• open literature environments
• LLM-based agents

Innovation

• agentic search
• taxonomy-guided benchmark
• academic paper retrieval
• open literature environment
• ScholarQuest