TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages

2025-02-16

AI Summary

Existing MMLU benchmarks lack native Turkic-language construction, suffering from machine translation artifacts and insufficient cultural adaptation—hindering fair NLU evaluation for low-resource Turkic languages. To address this, we introduce TUMLU, the first fully natively authored multilingual multitask understanding benchmark for Turkic languages: covering eight Turkic languages and eleven academic disciplines, with a human-verified compact variant, TUMLU-mini. TUMLU employs native-speaking subject-matter experts for item authoring, cross-lingual balanced sampling, and systematic evaluation across multiple foundation models (Claude, Gemini, GPT, LLaMA). Crucially, it achieves deep morphosyntactic and cultural alignment—eliminating translation-induced distortions. Experiments reveal substantial capability gaps in mainstream multilingual LMs on Turkic languages. All data and open-source evaluation tools are publicly released, establishing a new standard for rigorous, culturally grounded NLU assessment in low-resource settings.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Being able to thoroughly assess massive multi-task language understanding (MMLU) capabilities is essential for advancing the applicability of multilingual language models. However, preparing such benchmarks in high quality native language is often costly and therefore limits the representativeness of evaluation datasets. While recent efforts focused on building more inclusive MMLU benchmarks, these are conventionally built using machine translation from high-resource languages, which may introduce errors and fail to account for the linguistic and cultural intricacies of the target languages. In this paper, we address the lack of native language MMLU benchmark especially in the under-represented Turkic language family with distinct morphosyntactic and cultural characteristics. We propose two benchmarks for Turkic language MMLU: TUMLU is a comprehensive, multilingual, and natively developed language understanding benchmark specifically designed for Turkic languages. It consists of middle- and high-school level questions spanning 11 academic subjects in Azerbaijani, Crimean Tatar, Karakalpak, Kazakh, Tatar, Turkish, Uyghur, and Uzbek. We also present TUMLU-mini, a more concise, balanced, and manually verified subset of the dataset. Using this dataset, we systematically evaluate a diverse range of open and proprietary multilingual large language models (LLMs), including Claude, Gemini, GPT, and LLaMA, offering an in-depth analysis of their performance across different languages, subjects, and alphabets. To promote further research and development in multilingual language understanding, we release TUMLU-mini and all corresponding evaluation scripts.

Problem

• Native MMLU benchmark for Turkic languages
• Addressing underrepresentation in linguistic evaluation
• Systematic evaluation of multilingual LLMs

Innovation

• Native language MMLU benchmark
• Turkic languages focus
• Manually verified dataset subset