MORE: A Multilingual Document Parsing Benchmark and Evaluation

2026-07-03

AI Summary

Existing document parsing benchmarks are predominantly limited to high-resource languages, making it difficult to assess the true performance of multilingual vision-language models on low-resource languages. To address this gap, this work introduces a large-scale multilingual document parsing evaluation benchmark covering 149 languages, offering the first comprehensive coverage of long-tail languages. Built from real-world documents, the benchmark includes complex structured elements such as code blocks, tables, and tables of contents. Its construction employs a hybrid data annotation pipeline combining model-assisted labeling with human refinement. This benchmark fills a critical void in multilingual document understanding evaluation and establishes a new baseline for assessing performance disparities of state-of-the-art models across diverse, realistic scenarios.

Abstract

Multilingual documents encapsulate rich regional cultures, scientific discoveries, and historical records. Parsing this content into structured, machine-readable formats is critical for unlocking global knowledge. However, existing benchmarks predominantly focus on high-resource languages like English and Chinese, creating an evaluation blind spot concerning model performance on other languages. While recent Vision-Language Models (VLMs) claim support for hundreds of languages, the lack of ground truth makes it impossible to empirically verify these capabilities. To bridge this gap, we introduce MORE, a large-scale benchmark designed for multilingual document parsing evaluation. MORE distinguishes itself through three key dimensions: (1) Unprecedented Scale: It covers 149 languages, making it the most linguistically diverse benchmark to date; (2) Structural Complexity: Unlike previous works, it extends evaluation beyond plain text to include structural elements such as code blocks, tables, and catalogs; and (3) Data Authenticity: All samples are curated from real-world documents via a model-assisted, human-refined annotation pipeline. We evaluate state-of-the-art models using MORE, establishing new performance baselines for long-tail languages and validating the benchmark's effectiveness in diagnosing model capabilities in realistic, diverse scenarios. The MORE dataset will be available at https://github.com/zimoqingfeng/MORE.

Problem

• multilingual document parsing
• evaluation benchmark
• low-resource languages
• Vision-Language Models
• structured content extraction

Innovation

• multilingual document parsing
• Vision-Language Models
• structured document understanding
• benchmark dataset
• low-resource languages