Making Failure Safe: A Constrained, Verifiable Agent Framework for Open-Web Data Collection

2026-06-25

AI Summary

Large language models (LLMs) often generate unreliable web scraping code due to dependency errors, brittle selectors, and page heterogeneity. To address this, this work proposes a typed JSON-based configuration paradigm that constrains LLM outputs to structured templates. By integrating a six-category scraper taxonomy, encapsulated utility functions, static Airflow DAG execution, and a rule-driven quality validation mechanism, the approach enables reusable, deterministic, and verifiable open-web data collection without requiring runtime LLM invocations. Evaluated across 80 independent tasks, the method achieves zero LLM calls during execution, the lowest average runtime, and consistently stable, efficient performance under repeated scheduling.

Abstract

LLMs and agents can generate web scrapers from natural-language requirements, but direct generation remains unreliable because of dependency errors, broken selectors, schema mismatches, and heterogeneous page structures. We propose a constrained, verifiable agent framework that shifts LLM output from free-form code to typed JSON collector configurations, combining a six-type collector taxonomy, template and utility-function constraints, static Airflow DAG execution, rule-based quality checking, and structured feedback correction. Experiments on 138 tasks show that the taxonomy supports description-based requirement typing, while confirming that stable instantiation requires completing source, field, and execution constraints beyond the initial description. On 80 independently source-verified tasks, the framework runs with zero execution-stage LLM tokens and the lowest average wall-clock time, trading moderate one-shot quality for a reusable, deterministic, and verifiable execution path suited to repeated scheduled collection. These results position the framework as a reusable, low-cost, and verifiable execution path for repeated open-web data collection.

Problem

• web scraping
• LLM reliability
• data collection
• schema mismatch
• page heterogeneity

Innovation

• constrained agent framework
• verifiable execution
• structured data collection
• LLM-generated scrapers
• Airflow DAG