Cross-Project Flakiness: A Case Study of the OpenStack Ecosystem

2026-02-10

AI Summary

Test flakiness severely undermines developer trust, CI reliability, and resource efficiency in large-scale software ecosystems, yet its cross-project manifestations remain underexplored. This study presents the first empirical analysis of test behavior across 649 projects in the OpenStack ecosystem, leveraging extensive CI logs, execution records, and a mixed-methods approach combining quantitative statistics with qualitative root-cause analysis. The work systematically identifies and quantifies two novel forms of flakiness: cross-project and inconsistent flaky tests. Findings reveal 1,535 cross-project and 1,105 inconsistent flaky tests, affecting 55% of the projects. Notably, 70% of unit tests are impacted by cross-project flakiness, significantly prolonging code review cycles and increasing resource consumption—challenging the long-held assumption that unit tests are inherently isolated.

Technology Category

• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Search and Optimization: Evaluation and Analysis
• Multiagent Systems: Multiagent Systems under Uncertainty

Application Category

• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

Automated regression testing is a cornerstone of modern software development, often contributing directly to code review and Continuous Integration (CI). Yet some tests suffer from flakiness, where their outcomes vary non-deterministically. Flakiness erodes developer trust in test results, wastes computational resources, and undermines CI reliability. While prior research has examined test flakiness within individual projects, its broader ecosystem-wide impact remains largely unexplored. In this paper, we present an empirical study of test flakiness in the OpenStack ecosystem, which focuses on (1) cross-project flakiness, where flaky tests impact multiple projects, and (2) inconsistent flakiness, where a test exhibits flakiness in some projects but remains stable in others. By analyzing 649 OpenStack projects, we identify 1,535 cross-project flaky tests and 1,105 inconsistently flaky tests. We find that cross-project flakiness affects 55% of OpenStack projects and significantly increases both review time and computational costs. Surprisingly, 70% of unit tests exhibit cross-project flakiness, challenging the assumption that unit tests are inherently insulated from issues that span modules like integration and system-level tests. Through qualitative analysis, we observe that race conditions in CI, inconsistent build configurations, and dependency mismatches are the primary causes of inconsistent flakiness. These findings underline the need for better coordination across complex ecosystems, standardized CI configurations, and improved test isolation strategies.

Problem

• test flakiness
• cross-project flakiness
• inconsistent flakiness
• Continuous Integration
• software ecosystem

Innovation

• cross-project flakiness
• inconsistent flakiness
• test reliability
• continuous integration
• empirical study