MetaHOPE: A Metaphor-Oriented Evaluation Framework for Analysing MT and LLM Translation Errors

2026-07-01

AI Summary

Metaphor translation poses significant challenges for machine translation and large language models due to its semantic complexity, contextual dependency, and cultural embeddedness. This work proposes MetaHOPE, a novel framework that introduces the first fine-grained human annotation scheme for assessing error severity in English–Chinese bidirectional metaphor translation and releases a high-quality bilingual parallel corpus with human-validated reference translations. Leveraging this resource, the study systematically evaluates the metaphor-handling capabilities of mainstream systems—including Google Translate, GPT-4 (correcting a likely typo from “GPT5.4”), and Hunyuan-7B—by integrating error categorization with post-editing techniques, thereby uncovering their characteristic failure patterns. The research establishes a new benchmark, provides open-source data, and lays a systematic foundation for future studies on metaphor understanding and cross-lingual generation.

Abstract

In this opinion paper, we propose MetaHOPE, an error severity-aware annotation framework for evaluating metaphor translations. Metaphors present challenges for machine translation (MT) and natural language understanding and processing (NLU, NLP), because it presents the features of semantic complexity, contextual dependency, and cultural embeddings that can lead to ambiguity issues for NLP models. To investigate how state-of-the-art NLP models perform on translating metaphors, we select three representative systems, i.e., GoogleMT, GPT5.4, and Hunyuan-7b as Neural MT (NMT) models and LLMs. We used two human-annotated metaphor corpora, including VUAMC and PSUCMC for English-to-Chinese and Chinese-to-English translation purposes. The original corpora we used are monolingual, where we carried out error annotation using the MetaHOPE framework, and also produced the human post-edited gold reference for bilingual use as a new resource. We believe the MetaHOPE evaluation framework for metaphor translation annotation, the parallel corpora resources, and the error analysis on SOTA automatic translation models can be useful and shed some light for the field of metaphor translation study. We share our resources publicly upon paper acceptance.

Problem

• metaphor translation
• machine translation
• large language models
• translation errors
• evaluation framework

Innovation

• metaphor translation
• error annotation framework
• severity-aware evaluation
• parallel metaphor corpus
• LLM translation analysis