GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scripts

2026-04-14

AI Summary

This work addresses the limited generalization of current OCR models beyond mainstream scripts, despite their strong performance on widely used writing systems. To this end, we introduce GlotOCR Bench, a comprehensive evaluation benchmark covering over 100 Unicode scripts, constructed from authentic multilingual text. The dataset includes both clean and degraded images rendered using Google Fonts, HarfBuzz, and FreeType, and supports bidirectional writing directions. We present the first systematic assessment of leading open- and closed-source OCR models on this benchmark, revealing that most models perform effectively on fewer than 10 scripts, with even the strongest failing to generalize reliably beyond 30. Models frequently produce noisy or character-confused outputs on unseen scripts. All code, data, and rendering pipelines are publicly released.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Optical character recognition (OCR) has advanced rapidly with the rise of vision-language models, yet evaluation has remained concentrated on a small cluster of high- and mid-resource scripts. We introduce GlotOCR Bench, a comprehensive benchmark evaluating OCR generalization across 100+ Unicode scripts. Our benchmark comprises clean and degraded image variants rendered from real multilingual texts. Images are rendered using fonts from the Google Fonts repository, shaped with HarfBuzz and rasterized with FreeType, supporting both LTR and RTL scripts. Samples of rendered images were manually reviewed to verify correct rendering across all scripts. We evaluate a broad suite of open-weight and proprietary vision-language models and find that most perform well on fewer than ten scripts, and even the strongest frontier models fail to generalize beyond thirty scripts. Performance broadly tracks script-level pretraining coverage, suggesting that current OCR systems rely on language model pretraining as much as on visual recognition. Models confronted with unfamiliar scripts either produce random noise or hallucinate characters from similar scripts they already know. We release the benchmark and pipeline for reproducibility. Pipeline Code: https://github.com/cisnlp/glotocr-bench, Benchmark: https://hf.co/datasets/cis-lmu/glotocr-bench.

Problem

• OCR
• Unicode scripts
• generalization
• multilingual
• benchmark

Innovation

• OCR benchmark
• script generalization
• multilingual OCR
• vision-language models
• Unicode scripts