A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends

2025-07-13

AI Summary

This paper presents a systematic survey of recent advances in multimodal large language models (MLLMs) for visually rich document understanding (VRDU). Addressing core challenges—including inadequate integration of textual, visual, and layout modalities, strong reliance on OCR outputs, and limited generalization and robustness—the work comprehensively analyzes MLLM architectures along three dimensions: (1) encoding-fusion mechanisms, (2) module-wise trainability, and (3) staged learning strategies. It comparatively examines OCR-dependent versus OCR-free paradigms and unifies pretraining, instruction tuning, and supervised fine-tuning techniques to propose a novel framework for joint multimodal feature modeling. The survey synthesizes benchmark datasets and representative models, identifies critical bottlenecks—such as layout-aware reasoning gaps and cross-domain instability—and outlines future directions toward efficient, generalizable, and robust VRDU systems. This work serves as a foundational theoretical reference and practical technical guide for the VRDU research community.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Visually-Rich Document Understanding (VRDU) has emerged as a critical field, driven by the need to automatically process documents containing complex visual, textual, and layout information. Recently, Multimodal Large Language Models (MLLMs) have shown remarkable potential in this domain, leveraging both Optical Character Recognition (OCR)-dependent and OCR-free frameworks to extract and interpret information in document images. This survey reviews recent advancements in MLLM-based VRDU, highlighting three core components: (1) methods for encoding and fusing textual, visual, and layout features; (2) training paradigms, including pretraining strategies, instruction-response tuning, and the trainability of different model modules; and (3) datasets utilized for pretraining, instruction-tuning, and supervised fine-tuning. Finally, we discuss the challenges and opportunities in this evolving field and propose future directions to advance the efficiency, generalizability, and robustness of VRDU systems.

Problem

• Surveying MLLM-based methods for visually rich document understanding
• Analyzing challenges in OCR-dependent and OCR-free document processing
• Exploring future directions for efficient and robust VRDU systems

Innovation

• MLLMs encode and fuse text, visual, layout features
• OCR-dependent and OCR-free frameworks for document understanding
• Pretraining, instruction-tuning, supervised fine-tuning strategies