Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource Languages

2026-07-02

AI Summary

This study addresses the challenges of using LLM-as-a-Judge in multilingual and low-resource language settings, where issues such as inconsistent evaluations, overreliance on model outputs, and insufficient human validation undermine reliability. Through a systematic analysis of 650 papers from the ACL Anthology—focusing deeply on 33 studies specifically addressing multilingual and low-resource contexts—the work reveals, for the first time, the systematic risks inherent in this evaluation paradigm for such languages. Combining literature review, cross-lingual empirical analysis, and consistency assessments, the study finds that existing approaches predominantly rely on single judge models and yield unstable results. To mitigate these limitations, the paper proposes targeted usage guidelines and evaluation criteria aimed at enhancing the reliability and fairness of LLM-based evaluation in multilingual environments.

Abstract

LLM-as-a-Judge has become the dominant evaluation paradigm for many natural language generation tasks, due to shortcomings of conventional metrics and high correlations with human judgment, albeit mostly in English. There are now attempts to extend LLM-as-a-Judge to multilingual settings including low-resource languages. However, LLMs have limited proficiency in low-resource languages, and there is often no adequate human validation in these settings. To highlight the scope of the problem and current practices, we explore the use of LLM-as-a-Judge evaluators in ACL Anthology papers focusing on multilingual settings and low-resource languages across a diverse set of tasks. Out of 650 papers mentioning LLM-as-a-judge, only 33 of them focus on low-resource or multilingual settings. Our in-depth analysis of these papers indicates inconsistent evaluation outcomes, a tendency to overtrust LLM judgments in multilingual settings, and the widespread reliance on a single judge model per study. To help the NLP community further, we conclude with recommendations about how to use LLM-as-a-Judge in multilingual and low-resource settings.

Problem

• LLM-as-a-Judge
• multilingual settings
• low-resource languages
• evaluation reliability
• human validation

Innovation

• LLM-as-a-Judge
• multilingual evaluation
• low-resource languages
• human validation
• evaluation reliability