Problem
Research questions and friction points this paper is trying to address.
document structure recognition
table-form documents
hierarchical structure
structural failure modes
benchmark evaluation
FormStruct-Bench:A Hierarchical and Diagnostic Benchmark for Table-Form Document Structure Recognition
🤖 AI Summary
This work addresses the limited diagnostic capability of existing benchmarks for table document structure recognition, which struggle to pinpoint specific failure modes. To overcome this, the authors introduce the first diagnostic benchmark targeting multi-level structural understanding—spanning document, schema, and component layers—constructed via a high-quality Director–Artist–Verifier annotation pipeline. The framework incorporates a hierarchical evaluation protocol and structure-specific diagnostic metrics, enabling precise attribution of recognition failures. It further provides a template-disentangled test set alongside analyses by difficulty and visual degradation slices. Experiments on 14 state-of-the-art systems reveal a stark gap: while document-level accuracy reaches up to 83.85%, fine-grained structural recognition scores remain below 18%, highlighting a significant disconnect between content reading and structural comprehension.
📝 Abstract
Transforming table-form documents into machine-processable records requires recovering not only their visible content but also the multilevel structure that organizes it. However, existing benchmarks evaluate either holistic document outputs or conventional table grids, and their aggregate scores provide little insight into where structural failures occur. We introduce FormStruct-Bench, a hierarchical and diagnostic benchmark that evaluates table-form document structure recognition at both the document level and progressively finer component levels, allowing aggregate performance to be traced back to specific structural failure modes. To construct auditable ground truth at scale, we annotate 70 reusable templates and expand them into 7,000 verified instances through a provenance-preserving Director--Artist--Verifier pipeline; all 1,100 instances in the template-disjoint test set additionally receive human review. Our evaluation protocol uses five primary metrics and three structure-specific diagnostics across page, schema, and component levels, together with slices over difficulty, structural constraints, and visual degradation. Across 14 API-hosted and locally deployable systems plus two SFT variants, the best document-level score reaches 83.85%, whereas the best reported fine-grained structural score remains below 18%. These results reveal a pronounced gap between reading document content and recovering the hierarchy and regional organization required for reliable table-form understanding.
Innovation
Methods, ideas, or system contributions that make the work stand out.
hierarchical evaluation
document structure recognition
diagnostic benchmark
provenance-preserving annotation
table-form understanding
🔎 Similar Papers
No similar papers found.
