cross-lingual / multilingual evaluation

This work identifies pervasive annotation errors in benchmark datasets used for evaluating multilingual large language models, with particularly severe quality deficiencies observed in French and Telugu test sets. To quantify their impact, the authors conduct meticulous human annotation and comparative analysis, systematically assessing multiple state-of-the-art models on both original and corrected versions of the datasets. Results reveal that annotation errors induce up to a 9.8% accuracy deviation—substantially distorting model capability assessments. Crucially, this study challenges the prevailing “immutable dataset” assumption in multilingual evaluation and proposes a paradigm shift toward versioned dataset management and continuous quality auditing. It further introduces a practical, systematic framework for dataset construction, validation, and iterative refinement. By establishing methodological foundations and actionable guidelines, this work advances the reliability and rigor of multilingual model evaluation.

This study addresses core challenges in multilingual benchmarking: English overrepresentation, data monopolization by high-resource countries, reliance on translation rather than localization, and misalignment with human judgment. We systematically analyze over 2,000 non-English benchmarks from 148 countries (2021–2024) via large-scale cross-lingual meta-analysis, Pearson correlation testing, and bias quantification across geographic and resource dimensions. Our first empirical finding shows that localized benchmarks exhibit significantly higher alignment with human evaluation (r = 0.68) than translated ones (r = 0.47). STEM-oriented tasks demonstrate strong correlation with human judgment (r = 0.70–0.85), whereas traditional tasks like XQuAD show weak correlation (r = 0.11–0.30). We propose a six-dimensional diagnostic framework for benchmark deficiencies, five key research directions, and three guiding principles—advancing a new paradigm of culturally adaptive, human-aligned multilingual evaluation.

This study addresses the scarcity of domain-specific data in low-resource language settings by systematically investigating strategies for constructing multilingual large language models (LLMs) as automatic evaluators (LLM-as-a-Judge). The authors conduct the first comprehensive experiments across high-, medium-, and low-resource conditions on English, Spanish, and Basque, examining the effects of instruction translation, monolingual versus multilingual supervised fine-tuning, model scale, and zero-shot versus fine-tuned paradigms. Their findings reveal that when in-domain data is available, fine-tuned smaller models can match the performance of closed-source large models; in its absence, large models under zero-shot settings are superior; and out-of-domain fine-tuning may even degrade evaluation performance. The work releases an expanded meta-evaluation dataset and codebase covering Spanish and Basque.

Widespread train-test overlap—often unreported—undermines the credibility and interpretability of language model evaluations. Method: We conduct the first systematic audit of disclosure practices across 30 major model developers, combining empirical data auditing, policy analysis, and developer interviews. Contribution/Results: Only 9 (30%) disclose overlap information—4 via open training data, 5 via statistics or methodology. Based on these findings, we propose a transparency reporting standard mandating public disclosure of either overlap statistics or representative training-data subsets. Our advocacy directly prompted three organizations to adopt critical new disclosures. This work has catalyzed community consensus on trustworthy evaluation benchmarks and established a methodological foundation and practical pathway for reproducible, verifiable large language model assessment.

Large language models (LLMs) exhibit degraded performance on low-resource programming languages (e.g., COBOL, Rust, Swift) due to insufficient training data. Method: This paper systematically investigates cross-lingual transfer learning, introducing the first large-scale empirical framework for characterizing transfer patterns across programming languages—evaluated across 11–41 languages and 1,808 task-language combinations on code completion, translation, and repair. Contribution/Results: We empirically identify Kotlin and JavaScript as optimal source languages; uncover task-specific, heterogeneous dependencies on source-language features—challenging natural-language transfer paradigms; and develop both a principled source-language selection guide and a feature-based prediction model. Our approach significantly improves performance on low-resource languages across diverse coding tasks, establishing a scalable methodology for legacy system modernization and AI support for emerging programming languages.

This work addresses the limited fine-grained diagnostic capability of existing large-scale multilingual evaluations, which hinders effective model optimization. The authors propose the first reusable multilingual agent-based diagnostic framework, decomposing post-evaluation analysis into five stages: planning, aggregation, instance inspection, cross-cultural reflection, and report generation. Integrating an expert knowledge base with multilingual understanding and culture-aware modules, the framework enables deep attribution across 33 model families, 11 benchmarks, 26 languages, and 34 cultural contexts. Leveraging an expert-driven diagnostic set comprising 54 queries across 15 languages, the approach translates scores into actionable guidance, yielding diagnostic reports that outperform the strongest baseline by 47% in quality and prevail in 87.9% of pairwise comparisons against human experts. The study further distills four key insights regarding deployment strategies, iterative refinement, and cross-cultural risk mitigation.

This study addresses cross-lingual scoring bias in existing automatic machine translation evaluation metrics, a problem hindered by the lack of parallel datasets with consistent quality annotations across languages. To overcome this limitation, the authors propose XQ-MEval, the first benchmark dataset enabling cross-lingual parallel quality evaluation. Built upon the MQM error taxonomy, the dataset is constructed by automatically injecting errors into high-quality reference translations and then filtering the resulting pseudo-translations through native speakers to ensure controlled quality levels, yielding source–reference–pseudo-translation triplets. Experiments across nine language directions reveal that nine widely used metrics consistently exhibit cross-lingual biases misaligned with human judgments. The paper further introduces a score normalization strategy that substantially improves fairness and correlation with human assessments in multilingual evaluation settings.

This work addresses the significant performance disparities exhibited by current large language models when executing identical tasks across different languages. To systematically evaluate this issue, the authors propose a cross-lingual algorithmic task benchmark that ensures linguistic parity, scalability, quantifiability, and transparency. The benchmark leverages templated generation of multilingual synthetic data and employs objective correctness metrics for evaluation. Experimental results demonstrate that state-of-the-art large language models consistently display pronounced cross-lingual performance gaps, thereby validating the effectiveness and necessity of the proposed benchmark in uncovering linguistic capability disparities inherent in these models.

This study addresses the challenges posed by noisy, non-parallel sentence pairs and low-quality translations prevalent in large-scale multilingual parallel corpora, as well as the absence of a unified, direction-aware evaluation framework. The authors decouple quality assessment into two distinct tasks: parallelism detection using multilingual embeddings and reference-free quality estimation employing reference-free evaluators. They further introduce a direction-aware evaluation routing mechanism to adaptively select appropriate assessment strategies. Comprehensive experiments on datasets such as FLORES-200 and BOUQuET evaluate four embedding models and nine quality estimators across diverse language directions. Results reveal that no single metric generalizes effectively across all directions, performance varies substantially by translation direction, naive ensembles dilute strong model signals, and evaluator scores correlate strongly with target-language coverage. The findings underscore the necessity of tailoring evaluation strategies to specific language directions.

This study systematically evaluates the generalization of commonsense knowledge in large language models across multilingual and multicultural contexts, with a particular focus on low-resource languages and underrepresented cultures. Building upon a human-curated extension of the BLEnD benchmark encompassing over 30 language–culture pairs, the evaluation features two tracks—short-answer and multiple-choice—and strictly enforces a zero-shot setting, prohibiting any training or fine-tuning on the benchmark data while allowing participation from any NLP system. As the first large-scale, purely evaluative benchmark for cross-cultural commonsense reasoning, the initiative attracted registrations from over 140 teams, with 62 submitting results. Analysis reveals that state-of-the-art approaches perform substantially worse on low-resource languages, highlighting critical challenges in cultural alignment and cross-cultural commonsense transfer.