How Far Can Machine Translation Quality Take You? Extrinsic Discourse Evaluation in Goal-Oriented Setups

2026-06-15

AI Summary

This study addresses the limitation of current machine translation evaluation, which predominantly focuses on intrinsic quality while neglecting the impact of translation errors on discourse coherence and multi-turn collaboration in downstream tasks. The work proposes the first extrinsic discourse-level evaluation framework grounded in a goal-oriented multi-agent interaction environment—the Diplomacy game—combining a static entity counting task with dynamic interaction scenarios to assess referential consistency and long-term coordination capabilities, respectively. Experimental results demonstrate that even high-intrinsic-quality machine translation systems frequently exhibit discourse-level referential errors, and such mistranslations significantly impair collaborative performance over multiple turns. These findings reveal that intrinsic evaluation metrics are insufficiently reliable for predicting real-world downstream task effectiveness.

Abstract

Existing machine translation (MT) metrics and discourse-focused evaluations primarily assess translation quality intrinsically, without measuring the downstream consequences of translation errors. In this work, we focus on extrinsic discourse evaluation of machine translation under two distinct regimes: static and interactive. Under the static regime, we propose an entity counting task as a probe of referential consistency in discourse. We show that high intrinsic MT quality does not reliably predict downstream discourse success and strong MT systems still produce referential inconsistencies. For the interactive regime, we study the goal-oriented multi-agent Welfare Diplomacy game as a probe of long-horizon communication and coordination. We find that interaction-specific translation failures impact downstream coordination. Our results highlight goal-oriented environments as a viable framework for discourse-sensitive extrinsic MT evaluation.

Problem

• machine translation
• extrinsic evaluation
• discourse
• goal-oriented
• referential consistency

Innovation

• extrinsic evaluation
• discourse consistency
• referential coherence
• goal-oriented communication
• machine translation metrics