Estimating Machine Translation Difficulty

2025-08-13

AI Summary

This paper introduces the novel task of *translation difficulty estimation*, aiming to automatically identify source-language texts that pose significant challenges for machine translation (MT) systems, thereby enhancing the discriminative power and rigor of MT evaluation. Methodologically, it formally defines translation difficulty as the predictability of MT output quality, proposes a family of deep learning models—Sentinel-src (including Sentinel-src-24 and Sentinel-src-25)—driven by multi-dimensional linguistic features, and designs dedicated evaluation metrics to quantify difficulty prediction performance. Empirical results demonstrate that our approach significantly outperforms heuristic rule-based and large language model–based baselines in prediction accuracy. The high-difficulty texts identified by Sentinel-src constitute a more challenging MT benchmark, effectively widening performance gaps among state-of-the-art MT systems. This work establishes a new paradigm for MT robustness research and difficulty-aware parallel data construction.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Evaluation and Analysis
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Machine translation quality has began achieving near-perfect translations in some setups. These high-quality outputs make it difficult to distinguish between state-of-the-art models and to identify areas for future improvement. Automatically identifying texts where machine translation systems struggle holds promise for developing more discriminative evaluations and guiding future research. We formalize the task of translation difficulty estimation, defining a text's difficulty based on the expected quality of its translations. We introduce a new metric to evaluate difficulty estimators and use it to assess both baselines and novel approaches. Finally, we demonstrate the practical utility of difficulty estimators by using them to construct more challenging machine translation benchmarks. Our results show that dedicated models (dubbed Sentinel-src) outperform both heuristic-based methods (e.g. word rarity or syntactic complexity) and LLM-as-a-judge approaches. We release two improved models for difficulty estimation, Sentinel-src-24 and Sentinel-src-25, which can be used to scan large collections of texts and select those most likely to challenge contemporary machine translation systems.

Problem

• Estimating difficulty in machine translation outputs
• Differentiating top-performing translation models effectively
• Identifying challenging texts for better evaluation benchmarks

Innovation

• Formalizing translation difficulty estimation task
• Introducing new metric for difficulty evaluation
• Developing Sentinel-src models for estimation