Toward Machine Translation Literacy: How Lay Users Perceive and Rely on Imperfect Translations

2025-10-10

AI Summary

This study investigates how fluency and adequacy errors in machine translation (MT) influence bilingual and non-bilingual users’ reliance behaviors in leisure contexts. Through an in-situ museum study integrating behavioral observation and surveys, we find that non-bilingual users—lacking metalinguistic awareness to assess translation quality—systematically overestimate MT reliability and exhibit excessive reliance; however, exposure to errors triggers reflective evaluation and adaptive adjustment of subsequent reliance strategies. Moving beyond traditional system-centric MT evaluation, the work reframes assessment around user cognitive support. It proposes a dual-path intervention: enhancing public translation literacy and integrating NLP interpretability techniques to strengthen users’ quality judgment capabilities. These findings provide empirical grounding and design implications for human-centered MT human–AI collaboration frameworks. (149 words)

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Humans and AI: Intelligent User Interfaces
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

As Machine Translation (MT) becomes increasingly commonplace, understanding how the general public perceives and relies on imperfect MT is crucial for contextualizing MT research in real-world applications. We present a human study conducted in a public museum (n=452), investigating how fluency and adequacy errors impact bilingual and non-bilingual users' reliance on MT during casual use. Our findings reveal that non-bilingual users often over-rely on MT due to a lack of evaluation strategies and alternatives, while experiencing the impact of errors can prompt users to reassess future reliance. This highlights the need for MT evaluation and NLP explanation techniques to promote not only MT quality, but also MT literacy among its users.

Problem

• Investigating how fluency and adequacy errors affect user reliance on machine translation
• Examining how bilingual and non-bilingual users perceive imperfect translations in casual settings
• Addressing non-bilingual users' over-reliance due to lack of evaluation strategies

Innovation

• Studied user reliance on imperfect machine translation
• Investigated fluency and adequacy error impacts
• Promoted machine translation literacy techniques