Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding

2025-10-16

AI Summary

Document understanding faces three key challenges: (1) OCR systems discard structural information; (2) multimodal large language models (MLLMs) exhibit limited contextual modeling capacity; and (3) conventional RAG frameworks struggle with heterogeneous modalities—text, tables, charts, and layout. To address these, this paper proposes a comprehensive multimodal RAG framework for document understanding. Methodologically, it introduces a novel “domain–modality–granularity” three-dimensional taxonomy and integrates OCR, MLLMs, vector retrieval, graph neural networks, and agent-based coordination to enable cross-modal joint modeling and reasoning. It further incorporates graph-structured document representation and an agent-driven retrieval–reasoning closed loop, substantially improving fine-grained comprehension, retrieval efficiency, and system robustness. The work also surveys mainstream datasets, evaluation benchmarks, and real-world applications, identifies open challenges, and provides a holistic technical roadmap for future research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Document understanding is critical for applications from financial analysis to scientific discovery. Current approaches, whether OCR-based pipelines feeding Large Language Models (LLMs) or native Multimodal LLMs (MLLMs), face key limitations: the former loses structural detail, while the latter struggles with context modeling. Retrieval-Augmented Generation (RAG) helps ground models in external data, but documents' multimodal nature, i.e., combining text, tables, charts, and layout, demands a more advanced paradigm: Multimodal RAG. This approach enables holistic retrieval and reasoning across all modalities, unlocking comprehensive document intelligence. Recognizing its importance, this paper presents a systematic survey of Multimodal RAG for document understanding. We propose a taxonomy based on domain, retrieval modality, and granularity, and review advances involving graph structures and agentic frameworks. We also summarize key datasets, benchmarks, and applications, and highlight open challenges in efficiency, fine-grained representation, and robustness, providing a roadmap for future progress in document AI.

Problem

• Addressing limitations of OCR and MLLMs in document understanding
• Developing Multimodal RAG for holistic document retrieval and reasoning
• Surveying taxonomy, datasets, and challenges for document AI progress

Innovation

• Multimodal RAG for holistic document retrieval
• Taxonomy based on domain and modality
• Graph structures and agentic frameworks integration