Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite
Existing machine translation benchmark datasets commonly suffer from noise, structural deficiencies, and inconsistent quality, undermining the reliability of multilingual evaluation. This work proposes the first automated quality assessment framework that integrates structured corpus auditing, neural quality metrics (COMET), and fine-grained error analysis powered by large language models (LLMs) to comprehensively diagnose and refine the EU20 benchmark. By evaluating major translation systems—DeepL, Google Translate, and ChatGPT—and analyzing both reference-based and reference-free COMET scores, the study reveals a strong correlation between low COMET scores and high-accuracy errors (e.g., HellaSwag), while ARC emerges as relatively clean. The project releases a cleaned multilingual EU20 dataset, reproducible code, and a practical quality-prioritization guideline, establishing a new paradigm for constructing reliable multilingual benchmarks.