Replication Packages in Software Engineering Secondary Studies: A Systematic Mapping

2025-04-17

AI Summary

This study addresses the widespread lack of standardized and accessible replication packages in systematic reviews (SRs) within software engineering (SE). We conducted the first large-scale empirical investigation, analyzing 528 secondary studies published between 2013 and 2023 using systematic mapping, regression analysis, bibliometric methods, and manual metadata coding. Results show a significant decade-long increase in replication package reporting—reaching 50.6% in 2023—yet the overall adoption rate remains low (mean: 25.4%), and persistent archiving via DOI is alarmingly rare (only 29.1%). We present the first comprehensive inventory of replication packages for SE secondary studies, exposing critical gaps in transparency and reproducibility. Based on these findings, we propose a practice recommendation mandating the public release of replication packages with persistent identifiers (e.g., DOIs). This work provides empirically grounded evidence and actionable guidance to enhance research credibility and reproducibility in SE.

Technology Category

• Application Domains: Software Engineering
• Search and Optimization: Metareasoning and Metaheuristics
• Knowledge Representation and Reasoning: Knowledge Engineering

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Data transparency and provenance
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Context: Systematic reviews (SRs) summarize state-of-the-art evidence in science, including software engineering (SE). Objective: Our objective is to evaluate how SRs report replication packages and to provide a comprehensive list of these packages. Method: We examined 528 secondary studies published between 2013 and 2023 to analyze the availability and reporting of replication packages. Results: Our findings indicate that only 25.4% of the reviewed studies include replication packages. Encouragingly, the situation is gradually improving, as our regression analysis shows significant increase in the availability of replication packages over time. However, in 2023, just 50.6% of secondary studies provided a replication package while an even lower percentage, 29.1% had used a permanent repository with a digital object identifier (DOI) for storage. Conclusion: To enhance transparency and reproducibility in SE research, we advocate for the mandatory publication of replication packages in secondary studies.

Problem

• Evaluates reporting of replication packages in SE systematic reviews
• Analyzes availability trends of replication packages over 2013-2023
• Advocates mandatory replication packages to improve research reproducibility

Innovation

• Analyzed 528 secondary studies systematically
• Regression showed replication packages increasing
• Advocated mandatory replication packages publication