Expanding the WMT24++ Benchmark with Rumantsch Grischun, Sursilvan, Sutsilvan, Surmiran, Puter, and Vallader

2025-09-03

AI Summary

A long-standing lack of machine translation (MT) evaluation benchmarks exists for the six Romansh varieties—Rumantsch Grischun, Sursilvan, Sutsilvan, Surmiran, Puter, and Vallader—hindering progress in low-resource minority language MT. Method: We construct the first high-quality, multilingual parallel evaluation dataset covering all six varieties, built upon the WMT24++ framework. Professional human translators ensure precise cross-variety alignment and dialectal accuracy, with rigorous validation of inter-variety consistency. Contribution/Results: This benchmark fills a critical gap in MT evaluation for Swiss minority languages, enabling bidirectional German ↔ Romansh variety translation assessment. Experiments reveal that state-of-the-art MT systems exhibit limited performance on German→Romansh directions, particularly in generating dialect-specific forms; large language models fail to meaningfully alleviate this bottleneck. Our work provides a reproducible, extensible evaluation infrastructure for low-resource dialectal MT research.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

The Romansh language, spoken in Switzerland, has limited resources for machine translation evaluation. In this paper, we present a benchmark for six varieties of Romansh: Rumantsch Grischun, a supra-regional variety, and five regional varieties: Sursilvan, Sutsilvan, Surmiran, Puter, and Vallader. Our reference translations were created by human translators based on the WMT24++ benchmark, which ensures parallelism with more than 55 other languages. An automatic evaluation of existing MT systems and LLMs shows that translation out of Romansh into German is handled relatively well for all the varieties, but translation into Romansh is still challenging.

Problem

• Expanding machine translation benchmarks to include Romansh language varieties
• Addressing limited resources for Romansh machine translation evaluation
• Evaluating translation challenges between Romansh varieties and German

Innovation

• Expanding WMT24++ benchmark with Romansh varieties
• Creating human-translated references for six dialects
• Evaluating MT systems and LLMs performance