Can AI Agents Synthesize Scientific Conclusions?

2026-06-09

AI Summary

This study evaluates the ability of AI agents to synthesize reliable scientific conclusions from multiple sources of evidence in high-stakes domains such as health. To this end, the authors introduce SciConBench—the first large-scale, dynamic benchmark for open-domain scientific conclusion synthesis—and develop SciConHarness, a clean-room evaluation framework designed to eliminate data leakage. Their methodology decomposes conclusions into atomic facts and employs fact-level precision and recall as objective evaluation metrics. Experimental results reveal that even the best-performing agents achieve a fact-level F1 score of only 0.337, while mainstream consumer-grade models frequently generate incomplete or contradictory conclusions. These findings demonstrate that current evaluations significantly overestimate AI capabilities due to data leakage, underscoring that robust scientific reasoning remains a substantial challenge.

Abstract

Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions. Yet, their ability to do so in high-stakes domains such as health remains unclear. We introduce SciConBench, a large-scale live benchmark of 9.11K questions and expert-written conclusions from systematic reviews to evaluate open-domain scientific conclusion synthesis. The benchmark draws on an expert-validated automated evaluation pipeline that decomposes conclusions into atomic facts and measures correctness and comprehensiveness via factual precision and recall. To mitigate data leakage, we further introduce SciConHarness, a clean-room evaluation harness that equips agents with controlled web interaction to ensure valid measurement. Evaluating 8 frontier models and deep research agents, we find that factual quality remains low: under clean-room settings, the best agent achieves only a factual F1 of 0.337. Our clean-room setting consistently reduces performance relative to unconstrained evaluation, suggesting that leakage inflates estimates of models' true synthesis capabilities. Finally, we audit consumer-facing agents (e.g., Google AI Overview, OpenEvidence) and find they frequently generate incomplete and sometimes contradictory conclusions, even when the ground-truth answer is available. Overall, our results show that reliable synthesis of scientific conclusions remains an open challenge, and that clean-room evaluation is essential for assessing open-domain AI agents.

Problem

• scientific conclusion synthesis
• AI agents
• factual correctness
• clean-room evaluation
• systematic reviews

Innovation

• scientific conclusion synthesis
• clean-room evaluation
• fact decomposition
• systematic review benchmark
• data leakage mitigation