playwright

Web applications exhibit high interface dynamism and complex navigation and form interactions, posing significant challenges for end-to-end test case generation—particularly low coverage and poor robustness. To address this, we propose a navigation modeling approach that integrates screen transition graphs with large language models (LLMs) to enable high-precision path exploration. We further design a state-graph-based automated testing framework for conditional forms, supporting dynamic DOM parsing and interaction modeling. Additionally, we construct the first dedicated benchmark dataset for form-interaction testing. Experimental evaluation across diverse, complex web applications demonstrates substantial improvements in test coverage and path discovery accuracy—especially for branching navigation and conditional form-filling tasks. Our approach establishes a scalable, empirically evaluable paradigm for web application reliability testing.

This study addresses the lack of empirical evidence on the real-world impact of UI testing frameworks in CI/CD pipelines. Using GitHub API data collection, YAML configuration parsing, CI log metric extraction, and controlled time-series analysis across open-source repositories, we systematically quantify the integration patterns and effects of Selenium, Playwright, and Cypress within GitHub Actions workflows. Results show that UI testing significantly improves test pass-rate stability but increases mean build duration by 12% initially. Highly active repositories prefer Playwright—its built-in retry mechanism reduces flaky-test-induced pipeline interruptions by 35%. This work fills a critical gap in understanding UI testing’s practical implications in production CI/CD environments, providing data-driven insights for quality assurance strategy design and framework selection.

This work investigates whether large language models (LLMs) can synthesize executable, reusable Selenium Python scripts from natural-language goals by parsing HTML/DOM structures to enable end-to-end web automation. To this end, we introduce the first code-first, task-driven benchmark for web automation code generation—covering 7 major website categories and 681 multi-step tasks—and propose an end-to-end verification protocol integrating static analysis, sandboxed execution, result assertion, and security safeguards. Experimental evaluation shows that GPT-4o-Mini achieves a functional success rate of 96.8% across 2,636 runs (91.7% on average for simple tasks), yet fails entirely on complex workflows; critically, no model produces production-grade code. This study constitutes the first systematic assessment of LLMs’ program synthesis capability in realistic web environments, establishing a standardized testbed and empirically grounded benchmark for browser automation and AI agent research.

This work addresses the challenge of enabling large language models (LLMs) to autonomously interact with dynamic web environments—mimicking human behaviors such as clicking, scrolling, and typing—to retrieve information effectively. To this end, we propose BrowserAgent, an interactive agent framework that operates directly on real browsers via Playwright, bypassing static HTML-to-text conversion used by prior approaches (e.g., Search-R1, WebDancer). BrowserAgent incorporates an explicit memory mechanism to support long-horizon, multi-hop reasoning and employs a two-stage training strategy combining supervised fine-tuning (SFT) and rejection fine-tuning (RFT) to enhance generalization under low-data regimes. Evaluated on multi-hop question answering benchmarks—including HotpotQA, 2Wiki, and Bamboogle—BrowserAgent-7B achieves approximately 20% higher accuracy than Search-R1, demonstrating substantial gains over static-text-based paradigms.

This work proposes an AI-driven autonomous testing framework that addresses the fragility of traditional web test scripts—often susceptible to UI changes and timing issues—and their inability to support natural language–guided security testing. By leveraging a containerized, decoupled architecture, the framework automatically translates natural language instructions into robust web interaction scripts while integrating OWASP Top 10 security validations. It introduces a novel semantic mapping from natural language to security probes and enhances reliability through five key strategies: context-aware selector generation, intelligent wait injection, failure-based learning, and others. Experimental results demonstrate substantial improvements: script generation success rates increase from 55% to 93%, navigation failures decrease eightfold, timing-related race conditions are reduced by 80%, and test authoring time is cut by 75%. Furthermore, security testing achieves detection rates of 85% for authentication bypasses and 95% for input validation flaws, with a false positive rate below 12%.

This work addresses the limitations of large language models (LLMs) in generating logically correct and interactable GUI applications, such as games, where existing evaluation methods relying solely on test cases fail to capture correctness in interaction flows and state transitions. To bridge this gap, we propose PlayCoder—a repository-aware, multi-agent closed-loop framework that enhances functional correctness and semantic alignment through iterative code generation, interactive evaluation, and targeted repair. We introduce the first PlayEval benchmark and the Play@k metric, along with an LLM-driven PlayTester that automatically detects interaction logic errors and enables precise fixes for GUI-specific bugs. Experiments demonstrate significant performance gains across both open- and closed-source models, achieving up to 38.1% Exec@3 and 20.3% Play@3 improvements, while uncovering silent logical errors missed by conventional metrics.

This work addresses the fragmented evaluation of GUI, CLI, and coding capabilities in existing benchmarks, which fails to capture long-horizon, cross-interface coordination. The authors propose the first mixed-interface benchmark centered on real user requests, comprising 114 tasks executed in authentic Ubuntu desktop environments that require agents to integrate multimodal actions within a single execution trajectory. They introduce a novel trajectory-aware scorer that automatically evaluates performance by jointly analyzing file system states, screenshots, system logs, and action sequences, effectively detecting shortcut behaviors such as output fabrication or hard-coded responses. Experimental results reveal that even the best-performing model–runtime combinations achieve only a 41.2% pass rate, and conventional outcome-based scoring substantially overestimates actual agent capabilities.

This study addresses the high maintenance costs and poor stability of UI testing in large-scale enterprise applications by proposing the first multi-agent autonomous testing system empirically evaluated in a real-world enterprise environment. The system integrates large language models, LangGraph orchestration, the Playwright execution engine, and a RAG-augmented knowledge base, enhanced with runtime DOM analysis to enable goal-free functional discovery, dynamic coverage expansion, and self-repair of failing tests. Experimental results across ten UI scenarios demonstrate that the system discovers over 100 testable functionalities, achieves a 70% scenario-level repair convergence rate (averaging 3.4 iterative rounds), yet only 10% of scenarios succeed on the first attempt and 38% yield ineffective executions—highlighting how unconstrained autonomy can lead to semantic drift. The findings underscore constrained autonomy and human-in-the-loop collaboration as essential pathways toward reliable autonomous testing.

Current code-generation agents fail to meet functional requirements in over 70% of web application scenarios, primarily due to the absence of automated deployment, browser-level validation, and a feedback loop for iterative repair. This work proposes TDDev, a novel framework that establishes the first fully automated test-driven development (TDD) pipeline by integrating structured acceptance test generation, browser-interaction simulation for validation, and failure-triggered repair signals within a multi-agent collaborative architecture. Empirical evaluation demonstrates that TDDev improves generation quality by 34–48 percentage points, and user studies confirm it eliminates the need for manual intervention entirely. Furthermore, the study reveals that alignment between model generation style and TDD strategy is critical: misalignment not only nullifies performance gains but also inflates token consumption by up to 25-fold.