Can OCR-VLMs Read Devanagari? A Stress-Test Benchmark and Post-Correction Study

2026-06-28

AI Summary

This study addresses the lack of systematic evaluation of OCR systems on Indic scripts like Devanagari under realistic scanning conditions, where performance often degrades significantly. The authors introduce the first stress-test benchmark for Devanagari OCR, combining synthetically degraded and real-world printed documents, and evaluate ten OCR engines and vision-language models—including Qwen-VL, DeepSeek-OCR, and Gemini. Key contributions include uncovering that specialized OCR-VLMs are prone to catastrophic repetition errors under degradation; proposing median scores and catastrophe rates as more robust alternatives to mean-based metrics; establishing the first error taxonomy for Devanagari OCR; and introducing a byte-level ByT5 post-correction method. Experiments reveal that synthetic data substantially overestimates performance, with chrF++ scores in real scenarios spanning 76 points; among open-source models, Qwen3-VL-8B performs best, and ByT5 post-correction yields gains of 1.2–1.5 points for select systems.

Abstract

OCR systems, ranging from classical engines to specialised OCR vision-language models (OCR-VLMs) and frontier multimodal LLMs, report strong results on English and Chinese document benchmarks, yet their behaviour on Indic scripts is largely uncharacterised. We benchmark ten systems on Devanagari (Hindi): classical EasyOCR; open VLMs (Qwen2.5-VL-3B, Qwen3-VL-8B, olmOCR-7B); specialised OCR-VLMs (DeepSeek-OCR, Unlimited-OCR); and frontier closed models (Gemini 2.5 Flash, Claude Opus 4.7, GPT-5.5, Mistral OCR), across four synthetic degradation conditions and 300 real printed scans. We report four findings. First, on clean rendered text all ten cluster within chrF++ 91 to 98, so synthetic text does not separate them. Second, under degradation the specialised OCR-VLMs are the most fragile: DeepSeek-OCR suffers rare but catastrophic repetition failures (outputs up to 71 the reference length) that wreck its corpus mean even though its median is the best of any system, which is why we report median and catastrophic-rate instead of the mean. Third, on real scans nine of the ten systems collapse (EasyOCR falls from chrF++ 93.6 to 58.3) and the field spreads across a 76-point range, so synthetic renders badly overstate Devanagari quality. Fourth, strong English OCR does not predict Indic OCR: GPT-5.5 drops to chrF++ 58.5 (tying classical EasyOCR) and olmOCR-7B, the model behind olmOCR-Bench, falls to 40.5, while the open Qwen3-VL-8B (75.2, runnable on a single 24 GB GPU) beats GPT-5.5 and approaches Mistral; Gemini and Claude lead at 86.3 and 82.2. An error taxonomy separates surface errors (numerals, punctuation) from structural ones (conjuncts, matras, nukta), and a byte-level (ByT5) post-corrector improves a cheap engine on its own error distribution (chrF++ +1.2 to +1.5) but does not transfer across engines. We release the benchmark, code, and models.

Problem

• Devanagari
• OCR-VLMs
• Indic scripts
• robustness
• benchmarking

Innovation

• Devanagari OCR
• stress-test benchmark
• OCR-VLM robustness
• post-correction
• error taxonomy