Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss

2026-04-14

AI Summary

Existing multilingual evaluation benchmarks predominantly emphasize mathematical reasoning and factual recall, failing to adequately capture models’ genuine multilingual capabilities. This work proposes a reference-free evaluation method based on round-trip translation: a text is translated into a target language and then back into the source language, with semantic consistency between the original and reconstructed texts serving as a proxy for multilingual generation quality—eliminating the need for human reference translations or stronger judge models. Leveraging this approach, the authors construct Lost in Translation (LiT), a benchmark spanning major global languages, which effectively uncovers critical performance gaps of current multilingual large language models in real-world scenarios. Experiments demonstrate that the proposed metric correlates strongly with LMArena user ratings (ρ = 0.94), significantly outperforming existing evaluation paradigms.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Multilingual benchmarks guide the development of frontier models. Yet multilingual evaluations reported by frontier models are structured similar to popular reasoning and knowledge benchmarks, but across many languages. We show such benchmarks, and consequently multilingual evaluations, measure mathematical reasoning and factual recall, not multilingual proficiency. For example, thinking variants dramatically outperform instruct variants on these benchmarks, yet often perform worse on real-world multilingual tasks, such as LMArena. We propose a simple alternative: evaluate multilingual capability via round-trip translation. Given text in a source language, translate it to a target language and back; semantic gaps between the original and result expose failures in multilingual generation capabilities. Round-trip translation correlates almost perfectly (\r{ho} = 0.94) with user ratings on LMArena with our benchmark, requires no human reference translations, and does not require a more capable multilingual judge than tested models. Lastly, we introduce Lost in Translation (LiT), a challenging round-trip translation benchmark spanning widely spoken languages worldwide, for realistic evaluation of multilingual frontier models.

Problem

• multilingual benchmarks
• multilingual proficiency
• round-trip translation
• frontier models
• evaluation

Innovation

• round-trip translation
• multilingual evaluation
• Lost in Translation (LiT)
• semantic fidelity
• frontier language models