NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

2026-06-23

AI Summary

This study investigates whether AI coding agents can achieve original scientific discoveries beyond mere replication on real-world research problems. To this end, the authors introduce NatureBench, the first benchmark comprising 90 executable, cross-disciplinary tasks derived from papers published in the Nature family of journals, along with NatureGym—a standardized evaluation framework featuring containerized environments, automated build pipelines, and a strict prohibition on web search to address fragmentation and unreliable assessment in scientific task evaluation. Experimental results show that even the strongest current agents surpass the original papers’ state-of-the-art performance on only 17.8% of tasks, with successes largely attributable to method transfer rather than genuine scientific innovation. Primary failure modes include incorrect method selection and insufficient computational resources.

Abstract

We introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems. NatureBench is built on NatureGym, an automated pipeline that constructs a standardized, per-task containerized environment from a source paper, addressing the environment-fragmentation problem that has limited the credibility of prior agent-on-research benchmarks. Evaluating ten frontier agent configurations under a strict web-search-disabled protocol, we find that the strongest model surpasses SOTA on only 17.8% of tasks under the g>0.1 criterion. Analysis of method pathways reveals that agents succeed primarily through methodological translation, converting scientific tasks into familiar supervised prediction problems, rather than through genuine scientific invention. Failures are dominated by wrong method choice and insufficient compute budget, not by task misunderstanding. We release the benchmark, the NatureGym pipeline, and a public leaderboard with maintainer-side reproduction. Code: https://github.com/FrontisAI/NatureBench

Problem

• AI coding agents
• scientific discovery
• Nature-family papers
• benchmark
• reproducibility

Innovation

• NatureBench
• NatureGym
• AI coding agents
• scientific discovery
• containerized benchmark