Problem
Research questions and friction points this paper is trying to address.
translation errors
multilingual LLM evaluation
machine-translated benchmarks
evaluation reliability
cross-lingual accuracy
Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation
🤖 AI Summary
This study addresses underexamined translation errors in current machine translation benchmarks that may compromise the reliability and comparability of multilingual large language model (LLM) evaluations. It presents the first systematic quantification of the isolated impact of target-side translation errors on multilingual LLM assessment outcomes. The approach leverages an LLM-based evaluator to generate MQM-style error annotations, integrates the xCOMET-XXL quality estimation model, and employs controlled variable analysis while holding the correctness of English source prompts constant. Findings indicate that although automatically generated error annotations exhibit discrepancies compared to human judgments, translation errors nonetheless induce a statistically significant drop in model accuracy. This result validates the efficacy of automated error localization methods when applied to real-world translation benchmarks.
📝 Abstract
Machine-translated benchmarks are widely used to assess the multilingual capabilities of large language models (LLMs), yet translation errors in these benchmarks remain underexplored, raising concerns about the reliability and comparability of multilingual evaluation. We address two practical gaps: (i) how well automatic MQM-style error spans from LLM judges and a span-aware QE baseline (xCOMET-XXL) match expert human span annotations on benchmark translations, and (ii) how strongly translation errors (as opposed to source-side issues in the English original) explain accuracy drops on translated benchmarks. We find that span agreement is non-trivial on naturally occurring benchmark translations, and that target-side translation errors are consistently associated with measurable, percentage-point drops in translated accuracy even after controlling for English correctness and source-side anomalies.
Innovation
Methods, ideas, or system contributions that make the work stand out.
translation error
multilingual LLM evaluation
span-aware quality estimation
MQM-style error annotation
xCOMET-XXL
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
