BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation

2025-02-06

AI Summary

Existing machine translation evaluation datasets suffer from inadequate non-English language coverage, narrow domain and centrality scope, and susceptibility to data contamination—hindering fair and robust cross-lingual quality assessment. To address these limitations, we introduce MQM-Global, the first non-English–first, multi-centric, paragraph-level translation quality evaluation benchmark, covering 23 languages spoken by half the world’s population. Our methodology features: (1) a source-language–centric construction paradigm prioritizing non-English languages; (2) multi-centric sampling and contamination-resistant paragraph alignment; and (3) an open, multilingual parallel corpus co-construction framework. Through expert human annotation, multi-domain/multi-register sampling, linguistic centrality analysis, and cross-lingual consistency validation, MQM-Global significantly enhances representativeness and accessibility for low-resource and underrepresented languages. It establishes a new standard for general-purpose, equitable, and robust translation quality evaluation.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Evaluation and Analysis
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

This paper presents BOUQuET, a multicentric and multi-register/domain dataset and benchmark, and its broader collaborative extension initiative. This dataset is handcrafted in non-English languages first, each of these source languages being represented among the 23 languages commonly used by half of the world's population and therefore having the potential to serve as pivot languages that will enable more accurate translations. The dataset is specially designed to avoid contamination and be multicentric, so as to enforce representation of multilingual language features. In addition, the dataset goes beyond the sentence level, as it is organized in paragraphs of various lengths. Compared with related machine translation (MT) datasets, we show that BOUQuET has a broader representation of domains while simplifying the translation task for non-experts. Therefore, BOUQuET is specially suitable for the open initiative and call for translation participation that we are launching to extend it to a multi-way parallel corpus to any written language.

Problem

• Develops a universal translation quality evaluation dataset.
• Focuses on non-English languages for broader representation.
• Simplifies translation tasks for non-experts across domains.

Innovation

• Multicentric non-English dataset
• Broad domain representation
• Multi-way parallel corpus initiative