SciCoQA: Quality Assurance for Scientific Paper--Code Alignment

2026-01-19

AI Summary

Scientific papers and their code implementations often exhibit inconsistencies that threaten research reproducibility. This work presents the first systematic definition and annotation of paper–code discrepancy types, introducing SciCoQA—a dataset comprising 611 samples spanning domains such as artificial intelligence, physics, and quantitative biology. The dataset integrates real GitHub issues with synthetically augmented data to support a scalable framework for generating and detecting discrepancies. Evaluation using large language models reveals that even the most advanced model, GPT-5, identifies only 45.7% of real discrepancies, underscoring both the difficulty of the task and the necessity of this study.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

We present SciCoQA, a dataset for detecting discrepancies between scientific publications and their codebases to ensure faithful implementations. We construct SciCoQA from GitHub issues and reproducibility papers, and to scale our dataset, we propose a synthetic data generation method for constructing paper-code discrepancies. We analyze the paper-code discrepancies in detail and propose discrepancy types and categories to better understand the occurring mismatches. In total, our dataset consists of 611 paper-code discrepancies (81 real, 530 synthetic), spanning diverse computational science disciplines, including AI, Physics, Quantitative Biology, and others. Our evaluation of 21 LLMs highlights the difficulty of SciCoQA, particularly for instances involving omitted paper details, long-context inputs, and data outside the models'pre-training corpus. The best performing model in our evaluation, GPT-5, can only detect 45.7\% of real-world paper-code discrepancies.

Problem

• scientific reproducibility
• paper-code alignment
• discrepancy detection
• code implementation
• research integrity

Innovation

• paper-code alignment
• synthetic data generation
• reproducibility
• discrepancy detection
• scientific QA