ThaiOCRBench: A Task-Diverse Benchmark for Vision-Language Understanding in Thai

2025-11-06

AI Summary

Existing vision-language model (VLM) benchmarks severely underrepresent low-resource languages such as Thai—particularly in document structure understanding tasks. To address this gap, we introduce ThaiOCRBench, the first multi-task vision-language benchmark dedicated to Thai, comprising 2,808 samples across 13 document understanding tasks. Our contributions are threefold: (1) We establish the first Thai-specific OCR evaluation framework, enabling zero-shot assessment; (2) Through systematic evaluation, we expose critical weaknesses of mainstream VLMs in handwritten text recognition and fine-grained text extraction, identifying root causes including linguistic bias and structural misalignment; (3) Rigorous human annotation and error analysis demonstrate that closed-source models (e.g., Gemini) outperform open-source counterparts on complex scripts. The benchmark is fully open-sourced, providing a reproducible standard and actionable insights for advancing Thai document intelligence.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We present ThaiOCRBench, the first comprehensive benchmark for evaluating vision-language models (VLMs) on Thai text-rich visual understanding tasks. Despite recent progress in multimodal modeling, existing benchmarks predominantly focus on high-resource languages, leaving Thai underrepresented, especially in tasks requiring document structure understanding. ThaiOCRBench addresses this gap by offering a diverse, human-annotated dataset comprising 2,808 samples across 13 task categories. We evaluate a wide range of state-of-the-art VLMs in a zero-shot setting, spanning both proprietary and open-source systems. Results show a significant performance gap, with proprietary models (e.g., Gemini 2.5 Pro) outperforming open-source counterparts. Notably, fine-grained text recognition and handwritten content extraction exhibit the steepest performance drops among open-source models. Through detailed error analysis, we identify key challenges such as language bias, structural mismatch, and hallucinated content. ThaiOCRBench provides a standardized framework for assessing VLMs in low-resource, script-complex settings, and provides actionable insights for improving Thai-language document understanding.

Problem

• Addressing underrepresentation of Thai language in vision-language benchmarks
• Evaluating VLMs on diverse Thai text-rich visual understanding tasks
• Identifying performance gaps in low-resource script-complex settings

Innovation

• First Thai text-rich vision-language benchmark
• Evaluates 13 task categories with human annotations
• Identifies performance gaps in low-resource language settings