AI translation of literary texts is "fine", but readers still prefer human translations

2026-06-24

AI Summary

This study addresses the limitations of existing automatic evaluation metrics in capturing reader experience—particularly immersion and literary effect—in AI-generated literary translation. Proposing a reader-centered evaluation paradigm, the authors recruited 15 experienced readers to assess human and agent-based large language model (LLM) translations of 15 novels under two conditions: holistic immersive reading and fine-grained pairwise comparison. Results reveal that while readers generally deemed machine translations “acceptable,” they significantly preferred human translations at the segment level (522 out of 772 comparisons) yet struggled to reliably identify translation sources (only 17 out of 30 correctly identified). The study further uncovers a marked disconnect between reader preferences and automatic metrics, modulated by readers’ beliefs about translation origin. The accompanying LAIT dataset comprises 1K reader comments, 2K preference judgments, and 7.2K annotated segments.

Abstract

AI translation of literary works is increasingly common. While the content may be rendered adequately, we do not know enough about how readers experience it in terms of immersiveness and literary effect, aspects poorly captured by automatic machine translation metrics or human evaluation targeting fluency and adequacy. We ask 15 avid readers to compare recently published human translations (HT) to machine translations (MT) generated with an agentic large language model (LLM)-based pipeline, for 15 recent novels in French, Polish, and Japanese and translated into English. Readers evaluated approximately 8K-word excerpts in two conditions: immersive reading of the whole excerpt (30 comparisons) and close reading of 386 aligned HT-MT chunk pairs (772 comparisons), with two readers per book and in alternating order of presentation. Overall, readers find MT "fine", but prefer HT (slightly at excerpt-level 19/30, more clearly at chunk-level 522/772) for its ease, clarity, and immersive nature. Readers' highlights show that MT's quality varies more within one book than HT's does. Crucially, readers cannot reliably tell the two apart (17/30 guess correctly) and tend to prefer the version they believe to be human. Automatic metrics, including LLM-as-a-judge approaches, fail to recover reader preferences and favor MT. We release LAIT (Literary AI Translation), a reader-centered evaluation dataset with 1K reader comments, 2K judgments and preference ratings, and 7.2K span-level annotations, along with our evaluation protocol and supporting interface.

Problem

• literary translation
• reader experience
• immersion
• machine translation evaluation
• human preference

Innovation

• reader-centered evaluation
• literary machine translation
• agentic LLM pipeline
• immersive reading
• LAIT dataset