Visual Information Extraction from Documents via Classification-Guided Large Vision-Language Models

2026-07-21

AI Summary

This work addresses the challenge of information extraction from visually rich documents, where diverse layouts and real-world interferences—such as stamps and low contrast—hinder performance. Existing approaches typically rely on extensive annotated data and layout-specific training. To overcome these limitations, the authors propose a classification-guided large vision-language model framework that decouples document-type classification from content extraction and leverages in-context learning to dynamically construct prompts, enabling zero-shot information extraction across multiple document types. Built upon Qwen2.5-VL-7B, the method employs conditional computation to reduce task uncertainty. Evaluated on a real-world tendering dataset comprising 16 certificate categories, it achieves a zero-shot F1 score of 86.43%, outperforming a strong supervised baseline by 18.35 percentage points; with fine-tuning, F1 further improves to 93.65%, attaining a normalized edit distance of 0.93.

Abstract

Visual information extraction (VIE) from visually rich documents remains challenging due to high layout variability and real-world impairments. Existing methods typically rely on sequential OCR pipelines or end-to-end models requiring extensive labeled data and layout-specific training, limiting their scalability.We propose a classification-guided large vision-language model (LVLM) framework for multi-type VIE that achieves high accuracy with minimal supervision. The approach decouples document-type classification from content extraction and employs in-context learning (ICL)-based dynamic prompt engineering to inject task-specific knowledge, enabling robust zero-shot inference across diverse layouts. From a theoretical perspective, the proposed method can be viewed as a form of conditional computation that reduces task uncertainty and improves information efficiency during prompt-based inference. Evaluated on a real-world bidding dataset with 16 certificate types, our zero-shot method (based on Qwen2.5-VL-7B) outperforms a strong supervised baseline by 18.35 percentage points in F1-score (86.43\% vs. 68.08\%) and 0.23 in normalized edit distance (0.90 vs. 0.67). Optional domain-specific fine-tuning further improves performance to 93.65\% F1 and 0.93 NED, demonstrating superior robustness against seals, watermarks, and low contrast. The framework offers an efficient, scalable solution for complex document understanding in office automation. Code is available at https://github.com/FairmeHIT/Multi-VIE, and fine-tuned models at https://huggingface.co/fairme/Qwen2.5-VL-7B-SFT.

Problem

• Visual Information Extraction
• Visually Rich Documents
• Layout Variability
• Real-world Impairments
• Document Understanding

Innovation

• classification-guided LVLM
• in-context learning
• zero-shot visual information extraction
• dynamic prompt engineering
• conditional computation