SiniticMTError: A Machine Translation Dataset with Error Annotations for Sinitic Languages

2025-09-24

AI Summary

Low-resource Chinese dialects (e.g., Cantonese, Wu) lack fine-grained, error-annotated machine translation (MT) evaluation datasets. Method: We construct the first trilingual parallel dataset covering English→Mandarin/Cantonese/Wu, featuring three-tier human annotations—error span, error type (e.g., grammatical, semantic, dialect-specific), and severity level—performed by native speakers and validated via iterative feedback and inter-annotator agreement checks. Contribution/Results: This dataset fills a critical gap in low-resource dialect MT evaluation and, for the first time, reveals systematic cross-dialectal patterns in error distribution and severity. It serves as foundational infrastructure for error-aware MT model training and evaluation, enabling rigorous quality diagnostics and robustness optimization for low-resource language translation.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Evaluation and Analysis
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Despite major advances in machine translation (MT) in recent years, progress remains limited for many low-resource languages that lack large-scale training data and linguistic resources. Cantonese and Wu Chinese are two Sinitic examples, although each enjoys more than 80 million speakers around the world. In this paper, we introduce SiniticMTError, a novel dataset that builds on existing parallel corpora to provide error span, error type, and error severity annotations in machine-translated examples from English to Mandarin, Cantonese, and Wu Chinese. Our dataset serves as a resource for the MT community to utilize in fine-tuning models with error detection capabilities, supporting research on translation quality estimation, error-aware generation, and low-resource language evaluation. We report our rigorous annotation process by native speakers, with analyses on inter-annotator agreement, iterative feedback, and patterns in error type and severity.

Problem

• Addressing limited machine translation progress for low-resource Sinitic languages
• Providing error annotations for English-to-Sinitic machine translation outputs
• Creating resources for error detection and quality estimation in MT systems

Innovation

• Dataset with error span and type annotations
• Supports fine-tuning models for error detection
• Focuses on low-resource Sinitic language evaluation