Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrieval

2025-05-08

AI Summary

This work investigates how OCR-induced document quality variations impact the reliability of Retrieval-Augmented Generation (RAG) systems. We propose a semantic answer evaluation benchmark and conduct the first systematic comparison of OCR-dependent RAG (Nougat + Llama 3.2) versus end-to-end vision-embedding RAG (ColPali) across multi-quality scanned documents. Results show that OCR-RAG exhibits superior robustness across heterogeneous document quality distributions, whereas vision-RAG achieves high performance only within its training data distribution and suffers markedly reduced generalization. The study empirically delineates the practical trade-off between computational efficiency and semantic fidelity, offering evidence-based guidance for RAG architecture selection in production settings. Our core contributions are: (1) the first end-to-end semantic QA evaluation framework tailored for vision-RAG, and (2) a rigorous characterization of the key performance trade-offs between OCR-dependent and vision-embedding RAG paradigms.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Retrieval-Augmented Generation (RAG) has become a popular technique for enhancing the reliability and utility of Large Language Models (LLMs) by grounding responses in external documents. Traditional RAG systems rely on Optical Character Recognition (OCR) to first process scanned documents into text. However, even state-of-the-art OCRs can introduce errors, especially in degraded or complex documents. Recent vision-language approaches, such as ColPali, propose direct visual embedding of documents, eliminating the need for OCR. This study presents a systematic comparison between a vision-based RAG system (ColPali) and more traditional OCR-based pipelines utilizing Llama 3.2 (90B) and Nougat OCR across varying document qualities. Beyond conventional retrieval accuracy metrics, we introduce a semantic answer evaluation benchmark to assess end-to-end question-answering performance. Our findings indicate that while vision-based RAG performs well on documents it has been fine-tuned on, OCR-based RAG is better able to generalize to unseen documents of varying quality. We highlight the key trade-offs between computational efficiency and semantic accuracy, offering practical guidance for RAG practitioners in selecting between OCR-dependent and vision-based document retrieval systems in production environments.

Problem

• Compares vision-based and OCR-based document retrieval systems
• Evaluates performance on varying document qualities and unseen data
• Analyzes trade-offs between computational efficiency and semantic accuracy

Innovation

• Vision-based document embedding eliminates OCR errors
• Systematic comparison of vision-based and OCR-based RAG
• Trade-offs between computational efficiency and semantic accuracy