Coding-agents can replicate scientific machine learning papers

2026-07-02

AI Summary

This work addresses the challenge of reliably reproducing computational claims in scientific machine learning papers, which often lack systematic traceability. We propose Paper-replication, a novel workflow that models replication as a structured, goal-driven verification task. In this framework, each computational claim is treated as a verifiable objective, guiding an autonomous coding agent to reconstruct methods, execute experiments, trace evidential provenance, and undergo gated validation to ensure completeness. Integrating goal logging, method reconstruction, experimental execution, and verification checks, the framework was evaluated across four papers with twelve independent runs—all successfully passed the completion gate, and all 158 verification targets were empirically substantiated. This approach substantially enhances the auditability and reliability of computational reproducibility in scientific research.

Abstract

Scientific machine learning papers typically make computational claims, e.g., that the relative mean square error is less than 5% or that the 95% predictive credible interval covers the test data. A coding agent can be prompted to replicate those claims from paper materials alone, but the prompt does not by itself reliably preserve progress or check whether generated evidence supports the paper's claims. We introduce Paper-replication, a workflow that makes each selected paper claim a target with recorded evidence, and implement it as a coding-agent skill. The workflow makes the agent record those targets, reconstruct the paper's method, run computational experiments, link generated outputs to provenance and comparisons with the paper's claims, record where matched evidence appears in the replication report, and pass validation checks before completion. We evaluate Paper-replication on twelve independent runs across four scientific machine learning papers. All twelve workspaces pass the completion gate, and all 158 recorded targets are matched with report coverage. Even in this completed workspace state, repeated runs differ in how papers are divided into targets, in numerical fidelity to the source papers, in elapsed replication time, in the number of intermediate executions replaced before final evidence is accepted, and in the rules used to accept evidence. Paper-replication makes completion depend on workspace evidence and validation checks rather than on the agent's final message.

Problem

• scientific machine learning
• paper replication
• computational claims
• evidence validation
• coding agents

Innovation

• paper replication
• coding agent
• scientific machine learning
• computational reproducibility
• evidence-based validation