WebLists: Extracting Structured Information From Complex Interactive Websites Using Executable LLM Agents
Prior web agent research primarily focuses on navigation and transaction tasks, neglecting large-scale structured data extraction from complex, interactive websites. Method: We introduce WebLists—a benchmark comprising 200 real-world, enterprise-grade data extraction tasks across four commercial domains—requiring agents to jointly perform webpage navigation, dynamic configuration, and high-precision schema alignment. Existing LLMs and state-of-the-art (SOTA) agents achieve only 3% and 31% recall, respectively, revealing severe generalization limitations. To address this, we propose BardeenAgent: the first agent that synthesizes generalizable CSS selectors by leveraging HTML structural regularities; it integrates HTML structure-aware action modeling with programmable replay for executable, reusable automation. Contribution/Results: On WebLists, BardeenAgent achieves 66% recall—more than double the SOTA—and reduces per-output execution cost by 3×.