METATR: A Multilingual, Evolving Benchmark for Automatic Text Recognition
Current evaluations of text recognition models are largely confined to modern English printed text, failing to reflect model performance in real-world scenarios involving multiple languages and diverse layouts. To address this limitation, this work proposes METATR—an evolvable, multilingual automatic text recognition benchmark encompassing 29 languages, varied scripts, and complex layouts, featuring the first practice-oriented dynamic evaluation framework. METATR integrates publicly available document data, establishes a unified evaluation protocol, and incorporates standardized prompt engineering, text normalization strategies, and multidimensional metrics—including handwriting robustness and computational efficiency—to enable fair comparison between open-source and closed-source vision-language models. Experimental results demonstrate that while closed-source models generally exhibit greater stability, they still show significant performance variations across different scripts and layouts, thereby validating METATR’s effectiveness in revealing the practical capabilities of text recognition systems.