VERITAS: Towards a General-Purpose Replication Tool for Scientific Research

2026-07-02

AI Summary

The exponential growth of scientific publications has rendered manual replication prohibitively costly and inefficient, compounded by the absence of general-purpose automated tools for validating research claims. To address this challenge, this work proposes VERITAS—the first domain-agnostic framework for scientific replication. VERITAS leverages a CLI-based coding agent to automatically extract assertions from papers, execute described methodologies, repair code errors, and generate replication scores, repair logs, and executable code repositories through evidence comparison and importance-weighted evaluation. The framework accommodates interdisciplinary papers and associated code inputs, and when evaluated on 65 multidisciplinary papers from CORE-Bench and ReplicationBench, it significantly outperforms the Claude Code baseline under identical model and environmental conditions, achieving state-of-the-art performance across all metrics.

Abstract

AI tools are accelerating scientific publication while the systems that review it struggle to keep up, and independent verification of published research has become both harder and more important. As manual replication is slow and expensive, a growing line of work uses coding agents to automate parts of the process. Existing efforts are largely packaged as benchmarks with companion agents that only run inside the benchmark's own pipeline, and no general-purpose replication tool exists. We present VERITAS, a domain-agnostic replication framework built around CLI coding agents. Given a paper, a code repository, or both, VERITAS extracts the paper's claims, runs the methodology while resolving issues as they arise, and judges each claim against the evidence from experiment runs. The pipeline returns an importance-weighted Replication Score, a severity-rated log of every fix applied, and the patched codebase. We evaluate VERITAS on CORE-Bench and ReplicationBench, 65 papers spanning computer science, social science, medicine, and astrophysics. Against two strong Claude Code baselines on the same model and host environment, VERITAS achieves state-of-the-art performance and leads on every metric on both benchmarks.

Problem

• scientific replication
• general-purpose tool
• automated verification
• research reproducibility
• coding agents

Innovation

• general-purpose replication
• CLI coding agents
• automated verification
• Replication Score
• domain-agnostic framework