Co-Scraper: query-aware DOM Pruning and Reusable Scraper Synthesis for Lightweight Web Data Extraction

2026-06-12

AI Summary

This work addresses the challenge of efficiently generating lightweight, cross-page reusable data extractors from heterogeneous and complex web content. To this end, the authors propose a two-stage framework: first, query-aware DOM pruning is employed to simplify page structure by retaining only elements relevant to the target query; second, a fine-tuned Qwen3-8B large language model synthesizes stable, programmatic extraction strategies that generalize across pages. This approach uniquely integrates query-aware DOM pruning with large language model–driven extractor synthesis. Evaluated on the SWDE benchmark, it achieves an F1 score of 94.78% and a reusability success rate of 90.39%, substantially outperforming existing methods while demonstrating both high accuracy and strong generalization capability.

Abstract

The abundant and heterogeneous nature of web content necessitates automated information extraction, and generating scrapers that can be reused across similar web pages offers an effective solution for scalable data extraction. In this work, we propose Co-Scraper, a two-stage framework capable of handling the hierarchical complexity of long HTML documents. By integrating a query-aware DOM pruning mechanism with stable extraction strategy induction, Co-Scraper can effectively transforms web content into executable programmatic wrappers using a fine-tuned Qwen3-8B model. On the test set of SWDE, Co-Scraper achieves state-of-the-art performance with an F1 score of 94.78% and a reuse success rate of 90.39%. This framework significantly enhances the accuracy and resilience of data extraction, providing a highly efficient approach for web data acquisition tasks.

Problem

• web data extraction
• reusable scraper
• DOM pruning
• query-aware
• information extraction

Innovation

• query-aware DOM pruning
• reusable scraper synthesis
• programmatic wrappers
• web data extraction
• large language model