ExStrucTiny: A Benchmark for Schema-Variable Structured Information Extraction from Document Images

2026-02-12

AI Summary

Existing document understanding methods exhibit limited capability in fine-grained structured information extraction across diverse document types and varying schemas, and lack a unified benchmark. To address this gap, this work proposes the first comprehensive benchmark for structured information extraction from document images that supports flexible schemas and unified multi-task evaluation, integrating named entity recognition, relation extraction, and visual question answering across a wide range of document types and complex querying scenarios. A high-quality dataset is constructed through a combination of human annotation and synthetic data generation. The benchmark provides a systematic evaluation of state-of-the-art vision-language models with respect to schema adaptation, query ambiguity, and answer localization. Our analysis reveals key challenges faced by current models and offers a foundational resource for advancing general-purpose document understanding systems.

Technology Category

• Natural Language Processing: Information Extraction
• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Enterprise documents, such as forms and reports, embed critical information for downstream applications like data archiving, automated workflows, and analytics. Although generalist Vision Language Models (VLMs) perform well on established document understanding benchmarks, their ability to conduct holistic, fine-grained structured extraction across diverse document types and flexible schemas is not well studied. Existing Key Entity Extraction (KEE), Relation Extraction (RE), and Visual Question Answering (VQA) datasets are limited by narrow entity ontologies, simple queries, or homogeneous document types, often overlooking the need for adaptable and structured extraction. To address these gaps, we introduce ExStrucTiny, a new benchmark dataset for structured Information Extraction (IE) from document images, unifying aspects of KEE, RE, and VQA. Built through a novel pipeline combining manual and synthetic human-validated samples, ExStrucTiny covers more varied document types and extraction scenarios. We analyze open and closed VLMs on this benchmark, highlighting challenges such as schema adaptation, query under-specification, and answer localization. We hope our work provides a bedrock for improving generalist models for structured IE in documents.

Problem

• structured information extraction
• document images
• schema-variable extraction
• vision language models
• benchmark dataset

Innovation

• structured information extraction
• schema-variable IE
• document image understanding
• vision-language models
• benchmark dataset