Team ACK at SemEval-2025 Task 2: Beyond Word-for-Word Machine Translation for English-Korean Pairs

2025-04-29

AI Summary

This paper addresses the challenge of cross-cultural adaptation of culture-specific items (CSIs) and entity names in English–Korean translation of knowledge-intensive, entity-rich texts. Methodologically, it introduces the first fine-grained error taxonomy for English–Korean cross-cultural translation, systematically evaluating 13 large language models (LLMs) and machine translation (MT) systems via combined automatic metrics (BLEU, chrF) and bilingual native-speaker human evaluation, supplemented by error attribution and typological analysis. Key contributions include: (1) revealing a significant gap between automatic metric scores and human judgments of cultural appropriateness; (2) identifying systematic effects of entity type and frequency on translation quality; and (3) confirming that while LLMs outperform conventional MT overall, they still exhibit two critical bottlenecks—poor handling of low-frequency CSIs and inconsistent transliteration/naming of proper entities.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Translating knowledge-intensive and entity-rich text between English and Korean requires transcreation to preserve language-specific and cultural nuances beyond literal, phonetic or word-for-word conversion. We evaluate 13 models (LLMs and MT models) using automatic metrics and human assessment by bilingual annotators. Our findings show LLMs outperform traditional MT systems but struggle with entity translation requiring cultural adaptation. By constructing an error taxonomy, we identify incorrect responses and entity name errors as key issues, with performance varying by entity type and popularity level. This work exposes gaps in automatic evaluation metrics and hope to enable future work in completing culturally-nuanced machine translation.

Problem

• Transcreating English-Korean text to preserve cultural nuances
• Evaluating LLMs and MT models for entity translation accuracy
• Identifying gaps in automatic metrics for culturally-nuanced translation

Innovation

• Evaluating 13 LLMs and MT models
• Identifying errors via taxonomy construction
• Focusing on cultural adaptation gaps