ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

2026-05-28

AI Summary

This work addresses the critical lack of effective benchmarks for evaluating the true capabilities of AI systems in end-to-end autonomous scientific research. To this end, the authors introduce a comprehensive benchmark comprising 40 tasks derived from real scientific papers across 10 disciplines, providing source literature and raw data while withholding the target findings. The benchmark incorporates expert-designed multimodal scoring criteria and a unified evaluation protocol, implemented via a lightweight ResearchHarness framework to enable quantifiable and reproducible assessment of both autonomous research agents and native large language models. Experimental results reveal that even the strongest current system (Claude Code, scoring 21.5) and the best native model (Claude-Opus-4.7, scoring 20.7) fall significantly short of reliably reproducing scientific results, underscoring the benchmark’s essential role in measuring and advancing AI-driven scientific discovery.

Abstract

AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify. We present ResearchClawBench, a benchmark for evaluating autonomous scientific research across 40 tasks from 10 scientific domains. Each task is grounded in a real published paper, provides related literature and raw data, and hides the target paper during evaluation. Expert-curated multimodal rubrics decompose the target scientific artifacts into weighted criteria, enabling evaluation of target-paper-level re-discovery while leaving room for new discovery. We evaluate seven autonomous research (auto-research) agents under a unified protocol and seventeen native LLMs through the lightweight ResearchHarness. Current systems remain far from reliable re-discovery: the strongest autonomous agent, Claude Code, averages 21.5, and the strongest ResearchHarness LLM, Claude-Opus-4.7, averages 20.7, with an LLM frontier mean of only 26.5. Error analysis shows that failures concentrate in experimental protocol mismatch, evidence mismatch, and missing scientific core. ResearchClawBench provides a reproducible evaluation frontier for measuring progress toward autonomous scientific research.

Problem

• autonomous scientific research
• AI coding agents
• benchmark
• scientific discovery
• evaluation

Innovation

• autonomous scientific research
• benchmark
• AI coding agents
• multimodal rubrics
• ResearchClawBench