Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio

2026-06-15

AI Summary

Existing benchmarks lack real-world annotated data covering the full meta-analysis pipeline, limiting comprehensive evaluation of large language models (LLMs) in systematic scientific reasoning. This work introduces MetaSyn, the first expert-annotated benchmark encompassing research questions, PI/ECO criteria, 140,000 PubMed articles, positive and negative examples, and complete search strategies. It further proposes a stage-attribution evaluation framework. Leveraging RAG variants within a protocol-driven agent architecture, the system performs retrieval, screening, and synthesis under structured PI/ECO guidance. Experiments reveal that while retrieval achieves a 90.9% recall at K=200, end-to-end study inclusion recall drops to 52.7%, exposing a critical bottleneck in LLMs’ ability to accurately screen studies meeting precise PI/ECO criteria.

Abstract

Meta-analysis is a demanding form of evidence synthesis that combines literature retrieval, PI/ECO-guided study selection, and statistical aggregation. Its structured, verifiable workflow makes it an ideal substrate for evaluating systematic scientific reasoning, yet existing benchmarks lack ground truth across the full retrieval-screening-synthesis pipeline. We introduce MetaSyn, a dataset of 442 expert-curated meta-analyses from Nature Portfolio journals. Each entry pairs a research question with PI/ECO criteria, a retrieval corpus of 140k PubMed articles, verified positive studies, hard negatives that are topically similar but PI/ECO-ineligible, and complete search strategies and date bounds. Benchmarking twelve pipeline configurations (nine RAG variants and a protocol-driven agent) reveals a critical screening bottleneck: despite a retrieval ceiling of 90.9% recall at K=200, no system recovers more than 52.7% of ground-truth included literature. Current LLMs fail to reliably separate eligible studies from PI/ECO-failing distractors in pools of comparable topical relevance. Stage-attributed metrics capture where systems succeed and fail; a single end-to-end score does not.

Problem

• meta-analysis
• LLM agents
• evidence synthesis
• benchmarking
• study screening

Innovation

• meta-analysis
• LLM agents
• PI/ECO criteria
• systematic review benchmarking
• hard negatives