Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering

2026-07-08

AI Summary

This study systematically evaluates the robustness and cross-domain transferability of eight open-source vision-language models (VLMs) on visual question answering tasks involving diverse document types, including industrial documents, infographics, and slides. Through experiments under zero-shot, fully supervised fine-tuning, and few-shot settings, the work identifies visual layout parsing—not knowledge deficiency—as the primary bottleneck in document understanding. The findings reveal that large VLMs excel on structured documents but suffer significant performance degradation on complex layouts, whereas smaller models exhibit more pronounced improvements after fine-tuning, achieving effective cross-domain adaptation with as few as approximately 50 target-domain samples and even outperforming fully supervised baselines.

Abstract

Document Visual Question Answering (DocVQA) presents a complex multimodal challenge, requiring models to exploit visual, textual, and layout information from documents. Although Vision-Language Models (VLMs) have shown remarkable performance in text-vision tasks, their robustness and transferability to different document domains remains underexplored. In this study, we present a comprehensive evaluation of 8 open-source pretrained VLMs on DocVQA in three different document domains: industrial documents of varying type, infographics, and presentation slides. We systematically assess model performance under zero-shot evaluations, fully supervised finetuning with inter- and intra-dataset evaluations, and few-shot learning evaluations of knowledge transfer between domains. Our findings demonstrate that while large pretrained VLMs possess strong zero-shot baselines for structured layouts, their performance strongly decreases on visually complex layouts of infographics and slides. Although parameter scaling is a dominant factor on performance, supervised finetuning yields higher relative gains in smaller architectures. Furthermore, our cross-domain and few-shot experiments show that visual understanding is the main bottleneck for DocVQA, not a lack of knowledge from the VLMs. Using 50 target domain samples, the models finetuned in DocVQA with datasets of different domains rapidly adapt to the target domain documents, even surpassing their fully supervised counterparts in some cases.

Problem

• Document Visual Question Answering
• Vision-Language Models
• Domain Adaptation
• Multimodal Understanding
• Cross-domain Transfer

Innovation

• Domain-adapted VLMs
• Document Visual Question Answering
• Cross-domain evaluation
• Few-shot learning
• Visual understanding bottleneck