TWIX: Automatically Reconstructing Structured Data from Templatized Documents

2025-01-11

AI Summary

Existing approaches for structured information extraction from templated documents (e.g., reports, invoices) suffer from low efficiency, high cost, and reliance on handcrafted rules or expensive vision-language large models. Method: This paper introduces “template inversion,” a novel paradigm that jointly discovers document templates and localizes fields via end-to-end inverse inference of shared visual-structural templates. Our method integrates layout analysis, graph neural networks, and template-matching optimization to jointly model geometric layout, semantic content, and inter-field relationships. Contribution/Results: Evaluated on 34 real-world datasets, our approach achieves >90% average precision and recall—surpassing Amazon Textract, Azure Document Intelligence, and GPT-4-Vision by over 25%. It processes 817 pages 734× faster and at 1/5836 the cost. To our knowledge, this is the first method enabling efficient, generalizable extraction without manual rules or vision-language model inference.

Technology Category

• Natural Language Processing: Information Extraction
• Machine Learning: Structured Learning
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Web Mining and Content Analysis: Bridging structured and unstructured data

Abstract

Many documents, that we call templatized documents, are programmatically generated by populating fields in a visual template. Effective data extraction from these documents is crucial to supporting downstream analytical tasks. Current data extraction tools often struggle with complex document layouts, incur high latency and/or cost on large datasets, and often require significant human effort, when extracting tables or values given user-specified fields from documents. The key insight of our tool, TWIX, is to predict the underlying template used to create such documents, modeling the visual and structural commonalities across documents. Data extraction based on this predicted template provides a more principled, accurate, and efficient solution at a low cost. Comprehensive evaluations on 34 diverse real-world datasets show that uncovering the template is crucial for data extraction from templatized documents. TWIX achieves over 90% precision and recall on average, outperforming tools from industry: Textract and Azure Document Intelligence, and vision-based LLMs like GPT-4-Vision, by over 25% in precision and recall. TWIX scales easily to large datasets and is 734X faster and 5836X cheaper than vision-based LLMs for extracting data from a large document collection with 817 pages.

Problem

• Template-based Document
• Information Extraction
• Efficiency

Innovation

• Automated Information Extraction
• Template Recognition
• Efficiency and Accuracy