Problem
Research questions and friction points this paper is trying to address.
end-to-end GUI testing
visual-language models
bug discovery
web applications
Framework and Benchmark for Code-Driven Agentic Testing in Web Development
🤖 AI Summary
本文提出了一种名为CAT的代码驱动测试框架，通过编写Playwright代码来自动探索并发现Web应用中的错误，以解决现有评估方法在发现视觉-语言模型(VLMs)缺陷方面的不足。
📝 Abstract
End-to-end GUI testing is essential for verifying web applications, yet existing evaluations rely on predefined checklists and are confined to the data and frameworks of web generation benchmarks, leaving the bug-discovery ability of vision-language models (VLMs) systematically untested. We introduce \textbf{C}ode-driven \textbf{A}gentic \textbf{T}esting (CAT), a paradigm in which the agent writes Playwright code to drive the browser, gathers feedback, and autonomously explores web applications to uncover bugs. We instantiate CAT with CATJudge, an agentic framework that unifies Browser-Use and Computer-Use tools within a single environment and CATTest, a benchmark of 102 AI-generated web applications with carefully annotated bugs, built through close human-AI collaboration to feature complex interactions and subtle defects. Experiments with mainstream VLMs show that all evaluated models perform poorly, revealing a clear gap between current VLM capabilities and the demands of real-world testing in AI web development. We release our code and data at https://github.com/SleepyWithoutCoffee/CATJudge.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Code-Driven Agentic Testing
Playwright
CATJudge
CATTest
Vision-Language Models
🔎 Similar Papers
No similar papers found.
