SciNetBench: A Relation-Aware Benchmark for Scientific Literature Retrieval Agents

2025-12-16

arXiv.org

AI Summary

Existing scientific literature retrieval agents struggle to model complex inter-paper relationships—such as support, contradiction, or technological evolution—leading to fragmented knowledge and misinterpretation of research landscapes. This work introduces SciNet, the first systematically constructed benchmark dataset for relation-aware retrieval, encompassing 269 million papers and 8,940 tasks, and evaluates relational understanding at three granularities: ego-centric, pairwise, and path-wise. Built upon large-scale cross-disciplinary metadata and knowledge graphs, SciNet exposes a fundamental limitation of current mainstream retrieval methods, whose accuracy in modeling such relationships consistently falls below 20%. Incorporating this benchmark substantially improves the quality of downstream literature reviews by 25.3%, demonstrating the critical value of relation-awareness for scientific intelligence.

Abstract

The rapid development of AI agent has spurred the development of advanced research tools, such as Deep Research. Achieving this require a nuanced understanding of the relations within scientific literature, surpasses the scope of keyword-based or embedding-based retrieval. Existing retrieval agents mainly focus on the content-level similarities and are unable to decode critical relational dynamics, such as identifying corroborating or conflicting studies or tracing technological lineages, all of which are essential for a comprehensive literature review. Consequently, this fundamental limitation often results in a fragmented knowledge structure, misleading sentiment interpretation, and inadequate modeling of collective scientific progress. To investigate relation-aware retrieval more deeply, we propose SciNetBench, the first Scientific Network Relation-aware Benchmark for literature retrieval agents. Constructed from a corpus of over 18 million AI papers, our benchmark systematically evaluates three levels of relations: ego-centric retrieval of papers with novel knowledge structures, pair-wise identification of scholarly relationships, and path-wise reconstruction of scientific evolutionary trajectories. Through extensive evaluation of three categories of retrieval agents, we find that their accuracy on relation-aware retrieval tasks often falls below 20%, revealing a core shortcoming of current retrieval paradigms. Notably, further experiments on the literature review tasks demonstrate that providing agents with relational ground truth leads to a substantial 23.4% performance improvement in the review quality, validating the critical importance of relation-aware retrieval. We publicly release our benchmark at https://anonymous.4open.science/r/SciNetBench/ to support future research on advanced retrieval systems.

Problem

• scientific literature retrieval
• relational understanding
• scholarly relationships
• knowledge structure
• AI agents

Innovation

• relation-aware retrieval
• scientific literature
• AI agents
• knowledge networks
• SciNet