CMDR: Contextual Multimodal Document Retrieval

2026-07-07

AI Summary

Existing multimodal document retrieval methods often neglect cross-page context, rendering them ineffective for queries that require aggregating information across multiple pages. To address this limitation, this work proposes CMDR-Embed, a framework that jointly encodes multiple pages to produce a shared document-level contextual representation and derives context-aware page embeddings. We introduce CMDR, a novel multimodal retrieval task that explicitly requires modeling the holistic document context, along with CMDR-Bench, a dedicated benchmark for this task, and design a Contextual Multimodal Contrastive Learning (CMCL) objective. Experimental results demonstrate that CMDR-Embed significantly outperforms non-contextual baselines, underscoring the critical role of context-aware embeddings in multimodal document retrieval.

Abstract

Multimodal document retrieval aims to retrieve relevant pages while preserving both textual and visual content from the original document. However, existing benchmarks primarily evaluate simple lexical or semantic matching, and most methods encode pages independently. Consequently, they overlook the contextual information in the document required to resolve queries that aggregate information across multiple pages. In this paper, we introduce CMDR and CMDR-Bench, a new multimodal document retrieval task and benchmark that require modeling document context. To address this challenge, we propose CMDR-Embed, a contextual multimodal embedding framework that explicitly incorporates document context by jointly encoding multiple pages and deriving page-level embeddings from a shared contextual representation. Furthermore, we introduce CMCL, a contextual multimodal contrastive learning objective that effectively trains CMDR-Embed by balancing contextual modeling with page-level discriminability. Experiments demonstrate that CMDR-Embed significantly outperforms non-contextual embeddings, highlighting the importance of context-aware multimodal embeddings for advancing document retrieval.

Problem

• multimodal document retrieval
• document context
• contextual information
• cross-page query
• page-level embedding

Innovation

• contextual multimodal retrieval
• document context modeling
• CMDR-Embed
• multimodal contrastive learning
• page-level embedding