ColPali: Efficient Document Retrieval with Vision Language Models

2024-06-27

arXiv.org

AI Summary

Existing document retrieval systems rely on OCR followed by text embedding, resulting in lengthy pipelines, susceptibility to noise, and the neglect of critical visual cues—limiting performance in applications such as RAG. This paper proposes an image-native document retrieval paradigm that bypasses OCR entirely and performs end-to-end embedding directly on document page images. To this end, we introduce ColPali, a lightweight, trainable multimodal vision-language model integrating late-interaction matching with efficient image encoding. We further construct ViDoRe—the first benchmark dedicated to visual document retrieval. Experiments demonstrate that our approach significantly outperforms state-of-the-art methods on ViDoRe, achieves over 10× faster inference, and supports zero-shot cross-lingual and cross-domain page-level retrieval. All code, models, and data are publicly released.

Technology Category

• Computer Vision: Image and Video Retrieval
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Documents are visually rich structures that convey information through text, but also figures, page layouts, tables, or even fonts. Since modern retrieval systems mainly rely on the textual information they extract from document pages to index documents -often through lengthy and brittle processes-, they struggle to exploit key visual cues efficiently. This limits their capabilities in many practical document retrieval applications such as Retrieval Augmented Generation (RAG). To benchmark current systems on visually rich document retrieval, we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of various page-level retrieval tasks spanning multiple domains, languages, and practical settings. The inherent complexity and performance shortcomings of modern systems motivate a new concept; doing document retrieval by directly embedding the images of the document pages. We release ColPali, a Vision Language Model trained to produce high-quality multi-vector embeddings from images of document pages. Combined with a late interaction matching mechanism, ColPali largely outperforms modern document retrieval pipelines while being drastically simpler, faster and end-to-end trainable. We release models, data, code and benchmarks under open licenses at https://huggingface.co/vidore.

Problem

• Efficient retrieval of visually rich documents
• Overcoming limitations of text-based retrieval systems
• Enhancing performance in Retrieval Augmented Generation (RAG)

Innovation

• Vision Language Model for embeddings
• Late interaction matching mechanism
• End-to-end trainable document retrieval