Oops!... I did it again. Conclusion (In-)Stability in Quantitative Empirical Software Engineering: A Large-Scale Analysis

2025-10-08

AI Summary

This paper investigates validity threats arising from toolchain selection in quantitative empirical software engineering. We formally replicate three high-impact studies by extracting identical project data using four widely adopted mining tools—Git, JIRA, GitHub API, and BIC—and conduct both quantitative and qualitative comparative analyses. Results demonstrate that subtle technical discrepancies across tools—including data modeling assumptions, event definitions, and temporal window handling—propagate and significantly undermine consistency in baseline datasets, statistical outcomes, and ultimately research conclusions. To our knowledge, this is the first systematic study to reveal the critical impact of tool choice on the robustness of empirical findings. We propose a practical framework comprising enhanced tool reusability, improved analytical transparency, and mandatory cross-tool validation. This work advances methodological rigor in software evolution research by highlighting and mitigating tool-induced validity threats.

Technology Category

• Data Mining & Knowledge Management: Other Foundations of Data Mining & Knowledge Management
• Constraint Satisfaction and Optimization: Solvers and Tools
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Context: Mining software repositories is a popular means to gain insights into a software project's evolution, monitor project health, support decisions and derive best practices. Tools supporting the mining process are commonly applied by researchers and practitioners, but their limitations and agreement are often not well understood. Objective: This study investigates some threats to validity in complex tool pipelines for evolutionary software analyses and evaluates the tools' agreement in terms of data, study outcomes and conclusions for the same research questions. Method: We conduct a lightweight literature review to select three studies on collaboration and coordination, software maintenance and software quality from high-ranked venues, which we formally replicate with four independent, systematically selected mining tools to quantitatively and qualitatively compare the extracted data, analysis results and conclusions. Results: We find that numerous technical details in tool design and implementation accumulate along the complex mining pipelines and can cause substantial differences in the extracted baseline data, its derivatives, subsequent results of statistical analyses and, under specific circumstances, conclusions. Conclusions: Users must carefully choose tools and evaluate their limitations to assess the scope of validity in an adequate way. Reusing tools is recommended. Researchers and tool authors can promote reusability and help reducing uncertainties by reproduction packages and comparative studies following our approach.

Problem

• Investigates validity threats in software mining tool pipelines
• Evaluates tool agreement on data and research conclusions
• Analyzes how technical differences affect empirical study outcomes

Innovation

• Replicated studies with four independent mining tools
• Compared extracted data and analysis results quantitatively
• Evaluated tool agreement on conclusions and validity