🤖 AI Summary
This work proposes an intelligent web crawling approach based on multimodal large language models (MLLMs) to overcome the limitations of traditional crawlers, which struggle with dynamic, interactive websites and rely heavily on static HTML parsing and manual customization. The method integrates a specialized toolchain for web interaction and data extraction with a structured five-stage prompting mechanism, enabling fully automated, structured data collection from “index–content” architecture websites. By deeply coupling MLLMs with purpose-built tools, the system autonomously navigates complex user interfaces without human intervention. Experimental results demonstrate that the proposed approach significantly outperforms the Anthropic Computer Use baseline across six news websites and exhibits strong generalization capabilities in e-commerce scenarios.
📝 Abstract
Modern web scraping struggles with dynamic, interactive websites that require more than static HTML parsing. Current methods are often brittle and require manual customization for each site. To address this, we introduce Webscraper, a framework designed to handle the challenges of modern, dynamic web applications. It leverages a Multimodal Large Language Model (MLLM) to autonomously navigate interactive interfaces, invoke specialized tools, and perform structured data extraction in environments where traditional scrapers are ineffective. Webscraper utilizes a structured five-stage prompting procedure and a set of custom-built tools to navigate and extract data from websites following the common ``index-and-content'' architecture. Our experiments, conducted on six news websites, demonstrate that the full Webscraper framework, equipped with both our guiding prompt and specialized tools, achieves a significant improvement in extraction accuracy over the baseline agent Anthropic's Computer Use. We also applied the framework to e-commerce platforms to validate its generalizability.