AI Copilots for Reproducibility in Science: A Case Study

2025-06-25

AI Summary

Scientific reproducibility has long been hindered by opaque computational workflows, missing critical information, and prohibitively high replication costs. To address this, we propose OpenPub—a novel platform featuring a modular AI copilot architecture that systematically applies artificial intelligence to support open science tasks. Integrating natural language processing and code analysis, the architecture jointly parses papers, source code, and supplementary materials to extract structured knowledge and automatically generate executable Jupyter Notebooks. It further detects common reproducibility gaps—including omitted hyperparameters, undocumented preprocessing steps, and missing datasets. Evaluated on standard reproducibility benchmarks, OpenPub reduces average replication time from over 30 hours to approximately one hour, successfully reproducing most figures and core computational results. This work significantly enhances both the efficiency and transparency of computational reproducibility and establishes the first AI-driven infrastructure supporting end-to-end reproducibility in open science.

Technology Category

• Humans and AI: Other Foundations of Human Computation & AI
• Computer Vision: Computational Photography, Image & Video Synthesis
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Security and Privacy: Data transparency and provenance
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Open science initiatives seek to make research outputs more transparent, accessible, and reusable, but ensuring that published findings can be independently reproduced remains a persistent challenge. This paper introduces OpenPub, an AI-powered platform that supports researchers, reviewers, and readers through a suite of modular copilots focused on key open science tasks. In this work, we present the Reproducibility Copilot, which analyzes manuscripts, code, and supplementary materials to generate structured Jupyter Notebooks and recommendations aimed at facilitating computational, or "rote", reproducibility. We conducted feasibility tests using previously studied research papers with known reproducibility benchmarks. Results indicate that OpenPub can substantially reduce reproduction time - from over 30 hours to about 1 hour - while achieving high coverage of figures, tables, and results suitable for computational reproduction. The system systematically detects barriers to reproducibility, including missing hyperparameters, undocumented preprocessing steps, and incomplete or inaccessible datasets. These findings suggest that AI-driven tools can meaningfully reduce the burden of reproducibility efforts and contribute to more transparent and verifiable scientific communication. The modular copilot architecture also provides a foundation for extending AI assistance to additional open science objectives beyond reproducibility.

Problem

• AI platform enhances research reproducibility via modular copilots
• Reduces reproduction time from 30 hours to 1 hour
• Detects missing parameters, undocumented steps, and inaccessible data

Innovation

• AI-powered platform for open science tasks
• Generates Jupyter Notebooks for reproducibility
• Detects and reduces reproducibility barriers systematically