Should We be Pedantic About Reasoning Errors in Machine Translation?

2026-04-10

AI Summary

This study addresses the issue of inaccurate or logically inconsistent translations caused by flawed reasoning in machine translation systems. It presents the first systematic quantification of three types of reasoning errors across multilingual translation, introduces an automated annotation protocol, and designs a spectrum of intervention strategies—ranging from weak (e.g., deletion) to strong (e.g., re-reasoning, hindsight intervention, and Oracle-based correction)—to perturb reasoning trajectories and examine the relationship between reasoning faithfulness and translation quality. Experiments demonstrate that the proposed approach achieves high precision in identifying reasoning errors for Urdu, though its effectiveness is limited for Spanish. While strong interventions substantially improve error correction rates, they yield only marginal gains in overall translation quality, revealing that current models still exhibit weak reasoning faithfulness.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Across multiple language pairings (English $\to$ \{Spanish, French, German, Mandarin, Japanese, Urdu, Cantonese\}), we find reasoning errors in translation. To quantify how often these reasoning errors occur, we leverage an automated annotation protocol for reasoning evaluation wherein the goal is to detect if a reasoning step is any of three error categories: (1) source sentence-misaligned, (2) model hypothesis-misaligned, or (3) reasoning trace-misaligned. We probe the reasoning model with perturbed traces correcting for these identified reasoning errors using an array of weak-to-strong interventions: hedging, removal, re-reasoning after removal, hindsight, and oracle interventions. Experimenting with interventions on the reasoning traces suggests that small corrections to the reasoning have little impact on translation quality, but stronger interventions yield the highest resolution rates, despite translation quality gains being mixed. We find ultimately that reasoning errors in MT can be identified with high precision in Urdu but lower precision in Spanish, but that removing these reasoning errors does not resolve the initial errors significantly, suggesting limited reasoning faithfulness for machine translation.

Problem

• reasoning errors
• machine translation
• reasoning faithfulness
• error identification
• translation quality

Innovation

• reasoning errors
• machine translation
• automated annotation
• intervention analysis
• reasoning faithfulness