MMDocIR: Benchmarking Multi-Modal Retrieval for Long Documents

2025-01-15

AI Summary

Existing benchmarks lack comprehensive evaluation protocols for multimodal retrieval over long documents, particularly failing to systematically assess retrieval performance on layout-sensitive elements such as figures, tables, and mathematical formulas. Method: We introduce MMDocIR—the first large-scale multimodal long-document retrieval benchmark—featuring layout-level (beyond page-level) retrieval tasks. It comprises 1.6k expert-annotated and 173k self-bootstrapped weakly labeled queries. Our methodology incorporates a layout-structure-aware annotation framework, VLM-driven text encoding, OCR-text contrastive analysis, and multimodal fusion retrieval. Results: Experiments demonstrate that visual modalities yield significant performance gains; VLM-based text encoders outperform OCR-only encoders; and training on MMDocIR substantially improves model generalization. This work establishes the first fine-grained evaluation standard for multimodal document retrieval, bridging a critical gap in the field.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Multi-modal document retrieval is designed to identify and retrieve various forms of multi-modal content, such as figures, tables, charts, and layout information from extensive documents. Despite its significance, there is a notable lack of a robust benchmark to effectively evaluate the performance of systems in multi-modal document retrieval. To address this gap, this work introduces a new benchmark, named as MMDocIR, encompassing two distinct tasks: page-level and layout-level retrieval. The former focuses on localizing the most relevant pages within a long document, while the latter targets the detection of specific layouts, offering a more fine-grained granularity than whole-page analysis. A layout can refer to a variety of elements such as textual paragraphs, equations, figures, tables, or charts. The MMDocIR benchmark comprises a rich dataset featuring expertly annotated labels for 1,685 questions and bootstrapped labels for 173,843 questions, making it a pivotal resource for advancing multi-modal document retrieval for both training and evaluation. Through rigorous experiments, we reveal that (i) visual retrievers significantly outperform their text counterparts, (ii) MMDocIR train set can effectively benefit the training process of multi-modal document retrieval and (iii) text retrievers leveraging on VLM-text perform much better than those using OCR-text. These findings underscores the potential advantages of integrating visual elements for multi-modal document retrieval.

Problem

• Multimodal Document Retrieval
• Performance Evaluation
• Testing Standard

Innovation

• Multi-modal Document Retrieval
• MMDocIR
• Image-Text Fusion