Problem
Research questions and friction points this paper is trying to address.
machine translation
benchmark
evaluation methods
reliability
actionable assessments
Last Translation Benchmark
🤖 AI Summary
为解决现有翻译模型评估方法的局限性，提出Last Translation Benchmark，通过人工设计并同行评审的例子及手写验证规则来更可靠地评估模型。
📝 Abstract
For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, standard benchmarks for machine translation are approaching saturation. Further, automatic translation metrics are unreliable, vulnerable to reward-hacking, and provide unactionable assessments. Even gold human evaluation is not problem-free, because it often lacks reproducibility, objectivity, and scalability. Overall, this prevents us from tracking objective progress in the field and identifying pathways for improvement. We introduce the Last Translation Benchmark, a collection of human-authored and peer-reviewed examples (texts, images, audio, videos) that break leading machine translation models. We also present a new evaluation approach: each example comes with handcrafted verification rules describing concrete failure cases on that example, therefore allowing reliable and actionable future evaluation. The Last Translation Benchmark is a live dataset that accepts ongoing contributions. The latest version is LTBv1, containing accepted contributions prior to September 1st 2026, with future releases planned as new data is continuously collected.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Last Translation Benchmark
handcrafted verification rules
reliable and actionable evaluation
🔎 Similar Papers
No similar papers found.
