LangMark: A Multilingual Dataset for Automatic Post-Editing

2025-11-21

AI Summary

Existing automatic post-editing (APE) research is hindered by the scarcity of large-scale, multilingual, human-annotated datasets specifically constructed for neural machine translation (NMT) outputs. To address this gap, we introduce LangMark—the first large-scale, multilingual APE dataset comprising 206,983 source–machine-translated–post-edited triplets across seven languages. LangMark is designed to support few-shot prompting for large language models (LLMs) in APE tasks. Experimental results demonstrate that LLMs fine-tuned with only a small number of examples significantly outperform leading commercial translation systems, achieving substantial improvements in translation quality across multiple languages. By providing a standardized, diverse, and human-verified benchmark, LangMark fills a critical void in multilingual APE evaluation and serves as a foundational resource for training, evaluating, and advancing the generalization capabilities of APE models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Automatic post-editing (APE) aims to correct errors in machine-translated text, enhancing translation quality, while reducing the need for human intervention. Despite advances in neural machine translation (NMT), the development of effective APE systems has been hindered by the lack of large-scale multilingual datasets specifically tailored to NMT outputs. To address this gap, we present and release LangMark, a new human-annotated multilingual APE dataset for English translation to seven languages: Brazilian Portuguese, French, German, Italian, Japanese, Russian, and Spanish. The dataset has 206,983 triplets, with each triplet consisting of a source segment, its NMT output, and a human post-edited translation. Annotated by expert human linguists, our dataset offers both linguistic diversity and scale. Leveraging this dataset, we empirically show that Large Language Models (LLMs) with few-shot prompting can effectively perform APE, improving upon leading commercial and even proprietary machine translation systems. We believe that this new resource will facilitate the future development and evaluation of APE systems.

Problem

• Lack of large-scale multilingual datasets for automatic post-editing systems
• Correcting errors in machine-translated text to enhance translation quality
• Enabling automatic post-editing for English to seven target languages

Innovation

• Multilingual APE dataset with human annotations
• Few-shot prompting LLMs for automatic post-editing
• Dataset covers seven languages with NMT triplets