🤖 AI Summary
This work investigates whether large language models (LLMs) can synthesize executable, reusable Selenium Python scripts from natural-language goals by parsing HTML/DOM structures to enable end-to-end web automation. To this end, we introduce the first code-first, task-driven benchmark for web automation code generation—covering 7 major website categories and 681 multi-step tasks—and propose an end-to-end verification protocol integrating static analysis, sandboxed execution, result assertion, and security safeguards. Experimental evaluation shows that GPT-4o-Mini achieves a functional success rate of 96.8% across 2,636 runs (91.7% on average for simple tasks), yet fails entirely on complex workflows; critically, no model produces production-grade code. This study constitutes the first systematic assessment of LLMs’ program synthesis capability in realistic web environments, establishing a standardized testbed and empirically grounded benchmark for browser automation and AI agent research.
📝 Abstract
We introduce MacroBench, a code-first benchmark that evaluates whether LLMs can synthesize reusable browser automation programs from natural language goals by reading HTML/DOM and emitting Python with Selenium. MacroBench instantiates seven self-hosted sites: Airbnb-like, TikTok-like, Reddit-like, Instagram-like, Facebook-like, Discord-like, and Threads-like, covering 681 tasks across interaction complexity and targeting difficulty. Our end-to-end protocol validates generated code via static checks, sandboxed execution, and outcome verification including DOM assertions and database snapshots, and includes a safety suite for scraping, spam/abuse, and credential/privacy prompts. Across 2636 model-task runs, we observe stratified success: GPT-4o-Mini achieves 96.8 percent, GPT-4.1 achieves 95.3 percent, Gemini-2.5-Pro achieves 89.0 percent, and DeepSeek-V3.1 achieves 83.4 percent. Models handle simple tasks reliably at 91.7 percent but fail on complex workflows at 0.0 percent, and none meet production-quality coding practices despite functional completion. We release our complete benchmark pipeline, evaluation framework, and experimental results to enable reproducible assessment of macro synthesis for web automation.