How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMs

2024-10-24

arXiv.org

AI Summary

This study addresses the challenge of evaluating literary machine translation (MT) quality. We introduce LITEVAL-CORPUS—the first paragraph-level, human-validated parallel corpus for literary translation—covering four language pairs, 2,000+ translations, and 13,000+ sentences, enabling systematic comparison of human and large language model (LLM) output. Our key contributions include: (1) empirical evidence that the widely adopted Multidimensional Quality Metrics (MQM) framework exhibits severe miscalibration in literary contexts, whereas Best–Worst Scaling (BWS), a simple preference-ranking method, achieves 80–100% accuracy in identifying high-quality human translations; (2) student annotators applying MQM incur ~60% misclassification rates; (3) all automatic metrics attain ≤20% accuracy; and (4) human translations consistently surpass state-of-the-art LLMs in literariness and stylistic diversity, with LLMs exhibiting over-reliance on literal rendering and limited creative adaptation. The work establishes a robust, efficient human evaluation paradigm tailored to literary MT and provides a critical benchmark for future automatic evaluation research.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Evaluation and Analysis
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Recent research has focused on literary machine translation (MT) as a new challenge in MT. However, the evaluation of literary MT remains an open problem. We contribute to this ongoing discussion by introducing LITEVAL-CORPUS, a paragraph-level parallel corpus containing verified human translations and outputs from 9 MT systems, which totals over 2k translations and 13k evaluated sentences across four language pairs, costing 4.5k C. This corpus enables us to (i) examine the consistency and adequacy of human evaluation schemes with various degrees of complexity, (ii) compare evaluations by students and professionals, assess the effectiveness of (iii) LLM-based metrics and (iv) LLMs themselves. Our findings indicate that the adequacy of human evaluation is controlled by two factors: the complexity of the evaluation scheme (more complex is less adequate) and the expertise of evaluators (higher expertise yields more adequate evaluations). For instance, MQM (Multidimensional Quality Metrics), a complex scheme and the de facto standard for non-literary human MT evaluation, is largely inadequate for literary translation evaluation: with student evaluators, nearly 60% of human translations are misjudged as indistinguishable or inferior to machine translations. In contrast, BWS (BEST-WORST SCALING), a much simpler scheme, identifies human translations at a rate of 80-100%. Automatic metrics fare dramatically worse, with rates of at most 20%. Our overall evaluation indicates that published human translations consistently outperform LLM translations, where even the most recent LLMs tend to produce considerably more literal and less diverse translations compared to humans.

Problem

• Evaluating literary machine translation quality.
• Comparing human and LLM translation effectiveness.
• Assessing adequacy of human evaluation schemes.

Innovation

• Introduces LITEVAL-CORPUS for evaluation
• Compares human and LLM translations
• Uses BWS for accurate human translation identification