Can Retrievers Find the Same Paper from Different Aspects? A Multi-Aspect Full-Paper Scientific Retrieval Benchmark

2026-08-16

AI Summary

This study addresses the oversight of multi-dimensional query consistency in existing retrieval benchmarks by introducing Maple, the first expert-validated multi-dimensional full-text retrieval benchmark, alongside Maple-Synth, a synthetic query pipeline based on retrieval-augmented in-context learning. By integrating multimodal modeling, discussion mining, and multi-chunk aggregation retrieval, this work systematically evaluates models' capacity to consistently recall papers across diverse perspectives, including motivation and methodology. Experimental results demonstrate that while state-of-the-art models achieve 98.1% accuracy@20 on single dimensions, the all-dimension consistency rate drops precipitously to merely 15.7%. These findings reveal significant bottlenecks and challenges in current models regarding multi-faceted consistent retrieval, underscoring the necessity for more rigorous evaluation frameworks in academic search tasks.

Abstract

Scientific papers contain multiple searchable facets such as background, methods. However, many paper retrieval benchmarks merely evaluate individual query-paper relevance, while overlooking other facets of the same paper. To bridge this gap, we introduce MAPLE, an expert-validated benchmark for multi-aspect, full-paper retrieval that evaluates whether retrievers can consistently recover the same paper from queries targeting its motivation, method, and experimental findings. MAPLE contains 2,095 queries about recent ML and NLP papers, grounded in both textual and multimodal content. We further propose MAPLE-Synth, a retrieval-based in-context learning pipeline that leverages OpenReview discussions and human-written query exemplars to generate realistic queries reflecting researchers' interests in different aspects of a paper. Our expert validation shows that these queries are comparable in realism to human-written queries and highly relevant to the target papers. Experiments across lexical, scientific-domain, general-purpose text, and multimodal retrievers reveal a substantial gap between retrieving a paper from any one aspect and retrieving it from all aspects: the strongest model achieves 98.1% AnyAspect@20 but only 15.7% AllAspect@20. Experiment/result queries and table-referenced queries are particularly difficult across retrievers. Although multi-chunk aggregation improves multi-aspect paper retrieval, considerable failures persist. MAPLE provides a testbed for evaluating and developing retrievers that represent scientific papers more comprehensively.

Problem

• Multi-aspect retrieval
• Scientific paper retrieval
• Benchmark
• Full-paper retrieval

Innovation

• Multi-Aspect Retrieval
• Full-Paper Retrieval Benchmark
• MAPLE-Synth
• Retrieval-based In-Context Learning
• Multimodal Scientific Retrieval