MQM-Chat: Multidimensional Quality Metrics for Chat Translation

2024-08-29

International Conference on Computational Linguistics

AI Summary

Current chat translation evaluation lacks effective standards that simultaneously preserve conversational style and ensure contextual coherence. To address this, we propose MQM-Chat—the first multidimensional quality assessment framework specifically designed for conversational translation, grounded in the Multidimensional Quality Metrics (MQM) paradigm. MQM-Chat systematically defines chat-specific error types—including stylistic loss, over-correction, and misuse of internet slang—and introduces a style-sensitive annotation protocol alongside a context-aware scoring mechanism. The framework supports human–machine collaborative evaluation and is characterized by fine-grained error categorization, interpretability, and scalability. Empirical validation across five state-of-the-art translation models demonstrates that MQM-Chat accurately identifies both fundamental translation errors and model-specific deficiencies, significantly improving alignment between evaluation outcomes and model optimization objectives. Thus, MQM-Chat establishes a reliable benchmark for advancing chat translation quality research and iterative system development.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

The complexities of chats pose significant challenges for machine translation models. Recognizing the need for a precise evaluation metric to address the issues of chat translation, this study introduces Multidimensional Quality Metrics for Chat Translation (MQM-Chat). Through the experiments of five models using MQM-Chat, we observed that all models generated certain fundamental errors, while each of them has different shortcomings, such as omission, overly correcting ambiguous source content, and buzzword issues, resulting in the loss of stylized information. Our findings underscore the effectiveness of MQM-Chat in evaluating chat translation, emphasizing the importance of stylized content and dialogue consistency for future studies.

Problem

• Chat Translation Evaluation
• Dialog Coherence
• Chatty Characteristics Preservation

Innovation

• MQM-Chat
• Chat Translation Quality Assessment
• Naturalness and Coherence