Exploring the Impact of Integrating UI Testing in CI/CD Workflows on GitHub

2025-04-27

AI Summary

This study addresses the lack of empirical evidence on the real-world impact of UI testing frameworks in CI/CD pipelines. Using GitHub API data collection, YAML configuration parsing, CI log metric extraction, and controlled time-series analysis across open-source repositories, we systematically quantify the integration patterns and effects of Selenium, Playwright, and Cypress within GitHub Actions workflows. Results show that UI testing significantly improves test pass-rate stability but increases mean build duration by 12% initially. Highly active repositories prefer Playwright—its built-in retry mechanism reduces flaky-test-induced pipeline interruptions by 35%. This work fills a critical gap in understanding UI testing’s practical implications in production CI/CD environments, providing data-driven insights for quality assurance strategy design and framework selection.

Technology Category

• Humans and AI: User Experience and Usability
• Search and Optimization: Evaluation and Analysis
• Reasoning under Uncertainty: Causality

Application Category

• Responsible Web: Consent frameworks and practices on the web
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Background: User interface (UI) testing, which is used to verify the behavior of interactive elements in applications, plays an important role in software development and quality assurance. However, little is known about the adoption of UI testing frameworks in continuous integration and continuous delivery (CI/CD) workflows and their impact on open-source software development processes. Objective: We aim to investigate the current usage of popular UI testing frameworks-Selenium, Playwright and Cypress-in CI/CD pipelines among GitHub repositories. Our goal is to understand how UI testing tools are used in CI/CD processes and assess their potential impacts on open-source development activity and CI/CD workflows. Method: We propose an empirical study to examine GitHub repositories that incorporate UI testing in CI/CD workflows. Our exploratory evaluation will collect repositories that implement UI testing frameworks in configuration files for GitHub Actions workflows to inspect UI testing-related and non-UI testing-related workflows. Moreover, we further plan to collect metrics related to repository development activity and GitHub Actions workflows to conduct comparative and time series analyses exploring whether UI testing integration and usage within CI/CD processes has an impact on open-source development.

Problem

• Investigating UI testing framework adoption in CI/CD workflows on GitHub
• Assessing impact of UI testing on open-source development activity
• Analyzing UI testing integration effects in GitHub Actions workflows

Innovation

• Empirical study of UI testing in CI/CD
• Analyze GitHub repositories with UI testing
• Compare UI and non-UI testing workflows