🤖 AI Summary
This study addresses the high maintenance costs and poor stability of UI testing in large-scale enterprise applications by proposing the first multi-agent autonomous testing system empirically evaluated in a real-world enterprise environment. The system integrates large language models, LangGraph orchestration, the Playwright execution engine, and a RAG-augmented knowledge base, enhanced with runtime DOM analysis to enable goal-free functional discovery, dynamic coverage expansion, and self-repair of failing tests. Experimental results across ten UI scenarios demonstrate that the system discovers over 100 testable functionalities, achieves a 70% scenario-level repair convergence rate (averaging 3.4 iterative rounds), yet only 10% of scenarios succeed on the first attempt and 38% yield ineffective executions—highlighting how unconstrained autonomy can lead to semantic drift. The findings underscore constrained autonomy and human-in-the-loop collaboration as essential pathways toward reliable autonomous testing.
📝 Abstract
Maintaining reliable UI test suites in large-scale enterprise applications is a persistent and costly challenge. We present an industrial case study of a multi-agent autonomous testing system evaluated using anonymized execution data from a production-like enterprise UI testing prototype. The application features several hundred dynamic UI elements per screen. Built on a large language model with LangGraph orchestration, Playwright execution, and a RAG knowledge base, the system evolves from human-directed testing toward High-autonomy feature discovery and test execution: given no explicit test targets, it discovers over 100 testable features across 10 UI screens, dynamically expands coverage by an additional 15--30 features through runtime DOM analysis, and iteratively repairs failing tests without human intervention.
We analyzed 300 consecutive autonomous execution reports encompassing 636 individual test-case executions across 10 distinct scenario families. The system achieved a 70% repair convergence rate at the scenario-family level, with a mean of 3.4 repair iterations to convergence. However, only 10% of scenario families succeeded on first attempt, 38% of reports failed to produce any executable test artifact, and we documented concrete instances of assertion weakening and test-case deletion used as workaround mechanisms to achieve superficial convergence.
Our findings show that unrestricted autonomy leads to unstable and often misleading outcomes, while constrained autonomy transforms such systems into operationally viable workflows. Rather than advocating full autonomy, our findings suggest that reliable autonomous testing in enterprise-scale settings requires explicit constraints, validation boundaries, and human oversight to preserve semantic correctness and operational trustworthiness.