automation

This study systematically investigates how artificial intelligence (AI) hierarchically augments software testing—from zero to full automation—and identifies emergent AI-driven testing paradigms. Method: We propose AI4ST, the first ontology-based classification framework for AI-enhanced software testing, constructed via ontological engineering and integrating systematic literature review (SLR) with domain knowledge modeling. Contribution/Results: AI4ST provides a unified, structured taxonomy characterizing recent AI applications across test generation, execution, analysis, and maintenance. It precisely pinpoints critical technical gaps—including explainability, generalization capability, and test trustworthiness assurance—as well as interdisciplinary research opportunities. The framework establishes an extensible theoretical foundation and empirical analysis toolkit, thereby advancing the standardization and deep evolution of AI-driven software testing.

This study addresses the lack of empirical evidence on the real-world impact of UI testing frameworks in CI/CD pipelines. Using GitHub API data collection, YAML configuration parsing, CI log metric extraction, and controlled time-series analysis across open-source repositories, we systematically quantify the integration patterns and effects of Selenium, Playwright, and Cypress within GitHub Actions workflows. Results show that UI testing significantly improves test pass-rate stability but increases mean build duration by 12% initially. Highly active repositories prefer Playwright—its built-in retry mechanism reduces flaky-test-induced pipeline interruptions by 35%. This work fills a critical gap in understanding UI testing’s practical implications in production CI/CD environments, providing data-driven insights for quality assurance strategy design and framework selection.

This study addresses the imbalance in the test pyramid—characterized by an overreliance on coarse-grained integration and system tests, which leads to difficulties in fault localization and slow execution—by proposing, for the first time, a method to automatically generate unit tests from existing integration tests. The approach combines static and dynamic analysis to automatically isolate component dependencies and enhance coverage at the unit level. Implemented as a Node.js tool and evaluated on twelve open-source JavaScript projects, the technique produces high-quality unit tests that significantly improve test suite structure, thereby increasing both testing efficiency and maintainability.

To address test redundancy, high feedback latency, and inconsistent pre- vs. post-commit test selection objectives in large-scale multilingual monorepos, this paper proposes the first pipeline-aware, bi-objective reinforcement learning framework for regression test optimization: failure detection is prioritized during pre-commit testing, while flaky-change identification is emphasized post-commit. The method operates entirely on language-agnostic features, integrating pipeline semantic modeling with online log analysis to support dynamically evolving industrial test suites. Evaluated on 20 weeks of real-world CI data, it achieves significantly reduced average feedback latency, a 32% improvement in pre-commit test selection precision, and a 41% reduction in false positives—without requiring expensive features such as code coverage.

Continuous Integration (CI) practices suffer from severe monitoring deficiencies: developers largely neglect critical metrics such as “build health” and “time-to-fix failed builds,” while mainstream CI services offer only weak native monitoring capabilities, forcing reliance on fragmented and often redundant third-party tools. Method: We conducted a triangulated investigation—including documentation analysis, developer surveys, functional audits of CI platforms, and case studies of open-source projects—to systematically identify cognitive gaps and practical monitoring needs. Contribution/Results: Our study provides the first empirical evidence that although over 80% of developers track test coverage, only a minority monitor build health or timeliness; further, all major CI services lack built-in multidimensional monitoring support. These findings establish an evidence-based foundation for designing next-generation CI monitoring frameworks and prioritizing tooling enhancements.

This study presents the first empirical investigation into the evolution of CI/CD configurations in machine learning (ML) projects. Addressing the lack of understanding regarding how CI/CD configurations co-evolve with ML components, the authors analyze 508 open-source ML projects, 343 manually annotated commits, and 15,634 automated CI/CD commits. They propose a novel 14-category taxonomy capturing synergistic changes between CI/CD and ML components, develop a dedicated clustering tool to identify recurrent evolutionary patterns, and establish an empirically grounded model linking developer experience to CI/CD configuration modification behavior. Results show that 61.8% of CI/CD-related commits involve build strategy modifications; common anti-patterns—including dependency hardcoding and missing test frameworks—are identified; and senior developers modify CI/CD configurations more frequently and effectively than juniors, confirming the critical role of experience in CI/CD maintenance.

This study addresses the challenges of unstable end-to-end testing for Android applications in continuous integration (CI) due to fragile emulator configurations. It presents the first large-scale empirical analysis of 4,518 open-source projects, systematically examining how instrumentation tests are configured, how these practices evolve, and their comparative effectiveness in CI environments. Leveraging GitHub Actions metadata, the work evaluates three prevalent approaches: Gradle Managed Devices, community-reusable components, and custom scripts. Findings reveal that only 10.6% of projects adopt such testing; among them, community components demonstrate superior reliability and efficiency, third-party device labs are suitable for regression testing despite higher costs, and custom scripts, while flexible, suffer from high retry rates. The study thus illuminates current practices and critical trade-offs in Android CI testing.

This work addresses the challenge that rapid software development often compromises code maintainability, thereby hindering safe AI-assisted refactoring. To mitigate this, the authors propose an iterative refactoring approach that integrates large language models with human oversight. The method first leverages a code-specialized large language model to automatically generate high-coverage unit tests that capture existing program behavior. Subsequently, developers guide test-driven refactoring, while branch coverage metrics are used to constrain and validate model-generated outputs. Empirical evaluation demonstrates that the approach produces nearly 16,000 lines of reliable test code within hours, achieving up to 78% branch coverage on critical modules. This significantly reduces regression risk during large-scale refactoring and enhances the reliability and practicality of AI-assisted code restructuring.

This work addresses the challenge developers face in efficiently authoring CI/CD configurations due to limited DevOps expertise by proposing a large language model (LLM)-based, context-aware generation approach. The method leverages both natural language descriptions and repository structure to automatically produce accurate and executable pipeline configurations for platforms such as GitHub Actions and GitLab CI/CD. Integrated with automated validation and human-in-the-loop feedback mechanisms, this framework is the first to combine repository context understanding with natural language-driven configuration synthesis. Experimental results demonstrate that the approach significantly lowers the barrier to DevOps adoption, markedly improves the accuracy and validity of generated configurations, and substantially reduces manual configuration effort.

Existing program repair benchmarks inadequately reflect real-world repository-level continuous integration (CI) scenarios, as they overlook critical challenges such as non-code artifacts, environmental dependencies, and workflow constraints. This work introduces the first repository-level repair benchmark grounded in actual GitHub Actions executions, validating patches through faithful replay of original CI workflows. The benchmark includes 567 CI failures meticulously annotated into 12 fine-grained error categories. Innovatively adopting end-to-end CI workflow re-execution as the patch validation criterion, it enables error-type-aware evaluation. By integrating log analysis, fault localization, and large language model–generated candidate patches, the approach achieves strong performance on tool-enforced errors like formatting and static checks, attaining an overall best repair success rate of 18.9%, while environment- and configuration-related issues remain notably challenging.

This study addresses the challenges of regression testing in remote and hybrid work environments, where communication, coordination, and quality assurance are increasingly complex. Through qualitative interviews with 20 software practitioners, complemented by process analysis, tool integration assessment, and coding of collaborative practices, the research systematically investigates the sociotechnical evolution of regression testing in distributed settings. Findings indicate that while core testing phases remain largely stable, teams increasingly rely on documentation, automation, and integrated toolchains to sustain effectiveness. Standardized reporting formats, shared repositories, and traceability mechanisms significantly mitigate collaboration barriers inherent in remote work. The study offers novel insights and practical guidance for ensuring software quality in geographically dispersed development contexts.