AI-Assisted Computational Reproducibility on the FABRIC Testbed

2026-06-24

AI Summary

This work addresses the challenges of complex environment configuration and code adaptation in cross-domain scientific reproducibility by systematically integrating the large language model (LLM)-based programming assistant LoomAI into the FABRIC international testbed, establishing a novel human-AI collaborative paradigm for experimental replication. Emphasizing the reproducibility of scientific conclusions rather than merely numerical results, the approach supports diverse domains including MPI clusters, BBR congestion control, LAMMPS molecular dynamics simulations, and genomic analyses. In three interdisciplinary case studies, AI assistance reduced the manual effort required for reproduction by a factor of 4–6, while also exposing a critical bottleneck: current LLMs still require human intervention during analytical phases lacking structured workflows.

Abstract

Computational reproducibility remains difficult despite being central to scientific research. In this paper, we show how the international FABRIC testbed, combined with large language model (LLM) coding assistants through LoomAI, can simplify reproducing published experiments across multiple domains. We reproduced three case studies on FABRIC, covering BBR-family congestion-control evaluations, LAMMPS molecular dynamics scaling benchmarks on a CPU-only MPI cluster, and stress protein homeostasis genomics pipelines. Rather than focusing only on matching numerical outputs, we evaluate whether the reproduced experiments support the same scientific conclusions as the original studies. The AI assistant was effective in setting up the environment, adapting code, and debugging, but struggled with the analysis stages that lacked clearly defined workflows, which required human guidance to establish execution order and data dependencies. Across the case studies, the AI-assisted workflow reduced reproduction effort by roughly 4--6 times. We conclude with practical recommendations for improving AI-assisted reproducibility on research testbeds.

Problem

• computational reproducibility
• scientific reproducibility
• experiment reproduction
• research testbeds
• AI-assisted reproducibility

Innovation

• AI-assisted reproducibility
• FABRIC testbed
• large language models
• computational reproducibility
• scientific workflows