PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages

2026-07-07

AI Summary

This study addresses the significant bias in existing mathematical reasoning benchmarks toward high-resource languages, which impedes fair evaluation of large language models in medium- and low-resource settings. The authors present PluraMath, the first systematic extension of the PolyMath benchmark to 18 underrepresented languages spanning six language families. They employ a native-speaker-verified translation pipeline and a multilingual instruction-tuning evaluation framework, conducting fine-grained cross-lingual analyses across 27 reasoning models of varying scales. Their findings reveal substantial performance gaps between high- and low-resource languages, with model efficacy strongly correlated with instruction-following capability. The project releases the full dataset, toolchain, and evaluation framework to empower underrepresented language communities in advancing multilingual mathematical reasoning research.

Abstract

Mathematical reasoning has become a central task for evaluating and tuning reasoning Large Language Models (LLMs), yet existing benchmarks remain heavily biased toward high-resource languages, with English and Chinese dominating both pre-training corpora and evaluation suites. The recently released PolyMath (Wang et al., 2025) dataset represents a significant step forward, yet its coverage is still limited to 18 only high-resource languages. To address this gap, we introduce PluraMath, an extension of PolyMath to 18 additional {underrepresented languages spanning 6 language families -- ranging from mid-resource to extreme low-resource settings. We constructed the dataset through a human-curated pipeline, where native speakers thoroughly validated pre-computed translations. Using PluraMath, we then benchmark 27 reasoning LLMs across four model scales -- small, mid-size, large, and closed-source ensembles -- probing the multilingual mathematical reasoning capabilities of state-of-the-art models under diverse linguistic conditions. Our fine-grained analysis confirms a persistent gap in mathematical reasoning performance between high-resource and underrepresented languages, with stronger results largely associated with better instruction-following ability. We fully open-source our dataset, data acquisition pipeline, and evaluation framework, with the goal of lowering the barrier to multilingual benchmark development for underrepresented communities.

Problem

• mathematical reasoning
• low-resource languages
• multilingual evaluation
• language bias
• Large Language Models

Innovation

• multilingual mathematical reasoning
• low-resource languages
• PluraMath dataset
• human-curated translation
• LLM evaluation benchmark