🤖 AI Summary
Modern distributed software QA faces escalating quality costs due to high complexity, rapid iteration, and constrained resources. This paper conducts a systematic empirical evaluation of AI-based tools—including large language models (LLMs), AI agents, mutation testing, and static analysis—across exploratory testing, equivalence class analysis, test generation, test suite optimization, and end-to-end regression testing. It presents the first feasibility validation of AI agent–driven end-to-end regression testing and uncovers fundamental limitations of LLMs in semantic equivalence coverage, interpretability, and bias mitigation, proposing a complementary verification methodology. A proof-of-concept implementation demonstrates an 8.3% reduction in test execution volatility, yielding significant gains in efficiency and coverage breadth. Concurrently, the study identifies three critical deployment bottlenecks: poor interpretability, opaque decision-making (black-box behavior), and inconsistent semantic coverage.
📝 Abstract
Traditional quality assurance (QA) methods face significant challenges in addressing the complexity, scale, and rapid iteration cycles of modern software systems and are strained by limited resources available, leading to substantial costs associated with poor quality. The object of this research is the Quality Assurance processes for modern distributed software applications. The subject of the research is the assessment of the benefits, challenges, and prospects of integrating modern AI-oriented tools into quality assurance processes. We performed comprehensive analysis of implications on both verification and validation processes covering exploratory test analyses, equivalence partitioning and boundary analyses, metamorphic testing, finding inconsistencies in acceptance criteria (AC), static analyses, test case generation, unit test generation, test suit optimization and assessment, end to end scenario execution. End to end regression of sample enterprise application utilizing AI-agents over generated test scenarios was implemented as a proof of concept highlighting practical use of the study. The results, with only 8.3% flaky executions of generated test cases, indicate significant potential for the proposed approaches. However, the study also identified substantial challenges for practical adoption concerning generation of semantically identical coverage,"black box"nature and lack of explainability from state-of-the-art Large Language Models (LLMs), the tendency to correct mutated test cases to match expected results, underscoring the necessity for thorough verification of both generated artifacts and test execution results. The research demonstrates AI's transformative potential for QA but highlights the importance of a strategic approach to implementing these technologies, considering the identified limitations and the need for developing appropriate verification methodologies.