Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation

2026-06-14

AI Summary

This study addresses three key challenges in evaluating multilingual large language models: high annotation costs, difficulty in identifying translation errors, and conflation of culture-specific knowledge with general reasoning ability. To overcome these issues, the authors propose Multilingual-IRT, the first extension of Item Response Theory (IRT) to multilingual settings. This framework introduces language-specific difficulty offsets, decomposes discrimination parameters into content- and language-related components, and models residual language proficiency effects to enable efficient and accurate assessment. Evaluated on MMLU-Pro-X across 25 models and 29 languages, Multilingual-IRT reduces prediction error by 11–16%, successfully detects translation errors in all 28 non-English languages, and identifies culture-specific items that traditional evaluation methods overlook.

Abstract

Multilingual benchmarks are central to evaluating large language models (LLMs) across languages, but they suffer from three issues: exhaustive evaluation scales linearly with the number of languages, automatic translation introduces errors that are easily missed at scale, and some items conflate general and culture-specific knowledge. We address all three with a unified statistical framework, Multilingual-IRT, which extends Item Response Theory with per-language difficulty deviations, split discriminability separating content from language effects, and per-language ability residuals. Fitting Multilingual-IRT on 25 LLMs across 29 languages of MMLU-Pro-X, we show that its fitted parameters support three practical applications: predicting unobserved (item, LLM, language) instances with 11-16% lower binary cross-entropy than the strongest accuracy-based baseline, surfacing candidate translation errors distributed across all 28 non-English languages, whereas accuracy-based baselines concentrate detections in a few languages, and recovering culture-specific items that accuracy-based baselines miss.

Problem

• multilingual evaluation
• Item Response Theory
• translation errors
• culture-specific knowledge
• large language models

Innovation

• Multilingual-IRT
• Item Response Theory
• language-specific difficulty
• split discriminability
• culture-specific knowledge