VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA

2026-06-14

AI Summary

This work addresses the limitation of existing document-based question answering systems, which often neglect visual content such as charts and figures, thereby struggling to generate long-form answers that effectively integrate textual and visual information. To bridge this gap, we introduce a novel task requiring explicit references to visual elements within long answers and present VinQA, the first dataset designed to support this task. We explore two multimodal encoding strategies—full-page image encoding and separated text–image encoding—and fine-tune the Qwen2.5-VL multimodal large language model accordingly. Additionally, we propose M-GroSE, an evaluation framework featuring the Visual Source F1 metric. Experimental results demonstrate that our fine-tuned open-source model substantially narrows the performance gap with state-of-the-art closed-source models, and that full-page encoding, after training, matches the effectiveness of more complex separated encoding approaches while producing visually grounded references that are semantically appropriate and highly faithful.

Abstract

Real-world documents combine text with tables, charts, photographs, and diagrams arranged in diverse layouts, yet existing research on multimodal large language models (MLLMs) for document QA predominantly produces text-only responses, underutilizing these visual elements. We introduce VinQA, a dataset for long-form answer generation where cited visual elements are explicitly interleaved with their supporting text and grounded in relevant document pages. To support this task, we study two encoding methods for feeding raw document page images into an MLLM, along with their visual-element citation mechanisms: (1) Page Encoding, which directly encodes full-page images with bounding boxes of visual elements and treats these boxed regions as citable units; and (2) Modality Encoding, which parses each page to extract text and crop visual elements, encodes them separately, and uses these cropped elements as citable units. In our experiments, we propose M-GroSE, a multimodal evaluation framework extending GroUSE to assess answers along four dimensions: completeness, answer relevancy, faithfulness, and unanswerability. We additionally report Visual Source F1 to directly measure visual citation accuracy. Although proprietary frontier models still achieve the best overall scores on the VinQA test split, fine-tuning open Qwen2.5-VL models on the training split substantially improves their performance and narrows this gap. Modality Encoding is initially more robust for complex documents with long text, many visual elements, and diverse citation requirements. After training on VinQA, however, Page Encoding reaches a comparable level, competing effectively even without the explicit parsing used in Modality Encoding. Finally, Visual G-Eval, an MLLM-based judge, confirms that fine-tuned models insert visual elements at semantically appropriate positions with faithful supporting text.

Problem

• multimodal document QA
• visual elements
• long-form answer generation
• interleaved citation
• real-world documents

Innovation

• VinQA
• visual element citation
• multimodal document QA
• Page Encoding
• Modality Encoding