BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models

2025-02-11

AI Summary

Existing multilingual benchmarks primarily target fundamental language understanding tasks, failing to enable fair cross-lingual evaluation of large language models (LLMs) on advanced capabilities—such as instruction following, reasoning, long-context comprehension, and code generation. Method: We introduce MLMA, the first multilingual benchmark covering 17 languages and explicitly designed to assess these four core competencies. MLMA features a novel tripartite native-speaker collaborative annotation protocol to systematically model semantic shifts introduced by translation, alongside multidimensional task design and rigorous bias-mitigation procedures. Contribution/Results: Empirical evaluation reveals substantial, persistent cross-lingual performance gaps across mainstream LLMs—gaps that cannot be closed merely through model scaling. MLMA is publicly released to serve as a standardized, capability-oriented evaluation framework for multilingual LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Previous multilingual benchmarks focus primarily on simple understanding tasks, but for large language models(LLMs), we emphasize proficiency in instruction following, reasoning, long context understanding, code generation, and so on. However, measuring these advanced capabilities across languages is underexplored. To address the disparity, we introduce BenchMAX, a multi-way multilingual evaluation benchmark that allows for fair comparisons of these important abilities across languages. To maintain high quality, three distinct native-speaking annotators independently annotate each sample within all tasks after the data was machine-translated from English into 16 other languages. Additionally, we present a novel translation challenge stemming from dataset construction. Extensive experiments on BenchMAX reveal varying effectiveness of core capabilities across languages, highlighting performance gaps that cannot be bridged by simply scaling up model size. BenchMAX serves as a comprehensive multilingual evaluation platform, providing a promising test bed to promote the development of multilingual language models. The dataset and code are publicly accessible.

Problem

• Evaluate advanced multilingual LLM capabilities
• Introduce BenchMAX for fair cross-language comparisons
• Highlight performance gaps beyond model scaling

Innovation

• Multilingual evaluation benchmark
• Native-speaking annotators
• Translation challenge dataset