Training AI Scientists to Replicate Research

2026-08-13

AI Summary

This work addresses the challenges of scientific paper replication, which stem from the lack of standardized evaluation mechanisms and limited scientific reasoning capabilities in current AI systems. The authors propose Replica—an extensible task space for paper replication—and introduce Faraday, a 27-billion-parameter “AI scientist” that autonomously executes replication tasks through multi-agent collaboration and tool-augmented code generation. A low-noise evaluation framework based on automated scoring rules is integrated with reinforcement learning-based post-training to significantly enhance the system’s autonomy in long-horizon scientific exploration. Experimental results demonstrate that Faraday outperforms Claude Opus 4.8 and GPT-5.5 on held-out replication tasks and exhibits reasoning behaviors more aligned with scientific norms.

Abstract

The replicability of papers is a cornerstone of scientific knowledge, ensuring the reliability of existing results and providing a base for further experiments. The act of replication typically illuminates details that were previously underspecified, and thus requires similar hypothesis-driven exploration to open-ended research. In this work, we develop Replica, a scalable task space for paper replication. To provide reward signal, we introduce an auto-generated rubric-based judge that has low noise and agrees with human assessment of replication quality. We post-train Faraday, a 27B-parameter "AI Scientist" agent that leverages coding agents as tools, surpassing the performance of Claude Opus 4.8 and GPT-5.5 on held-out replication tasks. Qualitative analysis of individual rollouts reveals that Faraday adopts a more scientifically-principled approach. We believe that our results provide a stepping stone towards AI agents capable of long-horizon scientific innovation without requiring complex harnesses.

Problem

• replicability
• AI Scientist
• scientific replication
• research reproducibility
• paper replication

Innovation

• paper replication
• AI scientist
• rubric-based judge
• agent post-training
• scientific reasoning