Enhancing Large Multimodal Models in Key Information Extraction via Scene-Aware Document Synthesis

2026-07-05

AI Summary

This work addresses the performance limitations of locally deployable lightweight multimodal models in key information extraction due to insufficient high-quality supervised data. To overcome this, we propose the SAYRE framework, which employs a scene-aware document synthesis approach to automatically learn content and layout patterns from a small set of sample documents, generating synthetic training triplets comprising documents, structures, and annotations. SAYRE further incorporates an error-driven mechanism to selectively augment challenging samples, enabling template-free generation of structurally realistic and diverse training data. The synthesized data is used to fine-tune multimodal large language models such as Qwen3-VL. Experiments demonstrate that SAYRE significantly improves the performance of edge-deployed models on both constrained and open-category key information extraction tasks, with particularly notable gains for smaller models and complex document structures like tables and contractual clauses, thereby validating the efficacy of the proposed data synthesis strategy.

Abstract

Key Information Extraction (KIE) converts visually rich documents into structured data, but practical deployment remains challenging: strong performance often relies on costly on-server Large Multimodal Models (LMMs), while compact locally deployable models lack sufficient KIE supervision. We present SAYRE, a scene-aware document synthesis framework for generating scalable KIE training data without hand-crafted template design. Given a few exemplar documents, SAYRE captures category-specific content patterns and layout conventions to synthesize document-schema-annotation triples. It further introduces error-driven generation, which expands real-world failure cases into hard training examples while preserving their structural patterns. Experiments on constrained- and open-category KIE show that SAYRE consistently improves Qwen3-VL backbones and achieves the strongest overall performance among on-device LMMs. Data scaling experiments show an overall upward trend as more synthesized data is introduced, especially for smaller models and open-category extraction. Error analysis further shows that synthesized training reduces field-level errors by improving schema-aware extraction over dense tables, business identifiers, and contract clauses. These results establish scene-aware synthesis as an effective data-centric approach for improving practical multimodal KIE.

Problem

• Key Information Extraction
• Large Multimodal Models
• Document Synthesis
• On-device Deployment
• Training Data Scarcity

Innovation

• scene-aware synthesis
• key information extraction
• error-driven generation
• multimodal LMMs
• document layout modeling