How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbations

2026-06-24

AI Summary

This study addresses the lack of systematic evaluation of robustness against visual perturbations in current vision-language models (VLMs) on structure-sensitive OCR reasoning tasks. The authors introduce OCR-Robust, a benchmark comprising 812 samples from documents, charts, and other structured formats, augmented with five perturbation types at three intensity levels each. They propose a multi-dimensional evaluation framework—including clean accuracy, Robustness under Clean Reference (RCR), Worst-case Clean Reference (WCR), and Clean Robustness Index (CRI)—to conduct the first comprehensive assessment of 18 state-of-the-art VLMs and OCR+LLM pipelines. The findings reveal that high clean accuracy does not necessarily imply strong robustness; inputs involving charts and tables are significantly more vulnerable than plain documents, with some models exhibiting severe performance degradation under worst-case perturbations.

Abstract

Vision-language models (VLMs) have achieved strong performance on OCR-based benchmarks and increasingly focused on text-rich understanding, but their robustness under controlled visual degradation remains insufficiently understood. This gap is critical for OCR reasoning, where visual corruption can induce OCR errors and structural distortions, thereby introducing uncertainty into the reasoning task. To systematically study this problem, we introduce OCR-Robust, a benchmark designed for evaluating OCR reasoning robustness under visual perturbations. It contains 812 samples across two complementary subsets: OCR1.0, covering documents, scene text, receipts, handwriting, and mathematical content, and OCR2.0, focusing on charts, geometry diagrams, and tables. To enable efficient yet informative evaluation, we conduct a pilot study over 18 candidate perturbations and select 5 representative types at 3 severity levels each based on their impact and cross-model discriminability. We evaluate robustness using clean accuracy, Relative Corruption Retention (RCR), Worst-Case Retention (WCR), and a composite Corruption Robustness Index (CRI), and benchmark 18 models spanning proprietary systems, open-source VLMs, and OCR+LLM pipelines. Our results show that higher clean accuracy does not necessarily imply stronger robustness, and that models can suffer pronounced degradation in the worst case on OCR tasks that are sensitive to structure, and charts and tables are substantially more fragile than document-like inputs under perturbation.

Problem

• OCR reasoning
• vision-language models
• visual perturbations
• robustness
• text-rich understanding

Innovation

• OCR-Robust
• visual perturbations
• vision-language models
• robustness evaluation
• Relative Corruption Retention