Multimodal Graph RAG for Long-range Visually Rich Document Understanding

2026-06-27

AI Summary

This work addresses the challenge of achieving cross-page global semantic understanding and multi-hop reasoning in visually rich, lengthy documents—a limitation of existing approaches. To this end, we propose a retrieval-augmented generation framework grounded in multimodal knowledge graphs, which, for the first time, enables automatic construction of multimodal knowledge graphs tailored to visual documents. Our approach integrates graph neural networks with multimodal large language models to facilitate collaborative reasoning. Additionally, we introduce DLVQA, the first document-level visual question answering benchmark designed for evaluating global QA capabilities across entire documents. Experimental results demonstrate that our method significantly outperforms current multimodal retrieval-augmented generation (MMRAG) and knowledge graph-based approaches on multi-hop QA/VQA tasks and the DLVQA dataset.

Abstract

Multimodal large language models (MLLMs) are widely applied to visual document understanding. However, comprehending long documents remains an issue by the limited context window. Though recent multimodal retrieval-augmented generation (MMRAG) can address this challenge by retrieving relevant pages. It still struggles with the visual question answering (VQA) requiring holistic comprehension of a document. To cope with this, knowledge graph (KG) that summarizes global knowledge of a document can provide an effective solution. However, most existing LLM-based KG construction methods handle only the language modality, leaving the automatic creation of multimodal KGs (MMKGs) for visually rich documents largely unexplored. In this paper, we introduce a multimodal graph-based RAG approach to tackle this problem. Existing LLM-based KG methods evaluate the QA performance relying on indirect evidence such as comprehensiveness, diversity, empowerment, and so on. The lack of annotated datasets for comprehensive document-level VQA poses a significant challenge to effective model evaluation. To overcome this limitation, we also introduce a new benchmark, DLVQA (document-level VQA), which provides reference summaries and corresponding supporting facts for global document-level questions. Experimental results show that our approach outperforms existing MMRAG or KG-based approaches on multi-hop QA/VQA benchmarks and DLVQA.

Problem

• multimodal knowledge graph
• document-level VQA
• long-range document understanding
• visually rich documents
• retrieval-augmented generation

Innovation

• Multimodal Knowledge Graph
• Graph-based RAG
• Document-level VQA
• Visually Rich Document Understanding
• DLVQA