The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers

2026-06-15

AI Summary

This study addresses the longstanding reproducibility crisis and inadequate documentation practices in artificial intelligence by systematically analyzing 56,800 papers from five premier AI conferences between 2014 and 2024. Employing quantitative content analysis, rigorous data quality control, and a reproducibility inference model grounded in historical empirical rates, the work tracks the decade-long evolution of seven key indicators, including code and data sharing. The findings reveal, for the first time, a spontaneous and sustained improvement in open science practices across the community: code and data sharing rates rose from 11% to 64%, and overall reproducibility increased from 28% to 64%. Notably, this significant progress preceded the adoption of formal reproducibility checklists, suggesting that cultural shifts toward openness emerged prior to institutional interventions.

Abstract

The reproducibility crisis has directed the AI research community toward improving documentation practices. Several studies have identified methodological issues, and in response, the most impactful venues in the field have introduced reproducibility checklists. We seek to understand whether documentation practices have changed over time by assessing all published papers at five leading AI conferences over the past decade. Seven reproducibility variables were identified, quality-assured and used to analyse 56 800 publications. Our analysis reveals that in the period 2014 to 2024, documentation practices have improved; papers sharing both code and data increased nearly sixfold, from 11% to 64% Building on empirical reproducibility rates from a prior study, we estimate - inferred from documentation practices, not direct testing - that reproducibility increased from 28% in 2014 to 64% in 2024. Improvements in documentation practices predate the introduction of reproducibility checklists, suggesting these changes reflect a broader movement toward open science rather than a direct response to formal requirements.

Problem

• reproducibility
• open science
• AI research
• documentation practices
• reproducibility crisis

Innovation

• reproducibility
• open science
• documentation practices
• empirical analysis
• AI research trends