OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents

2026-08-09

AI Summary

This work addresses the limited linguistic scope of existing AI agent evaluation benchmarks, which predominantly focus on English and fail to capture performance in multilingual settings. We introduce OmnilingualGAIA2, an extension of the GAIA2 benchmark covering ten languages across five writing systems, incorporating machine translation validated by expert review and complemented by localized multilingual verifiers. Our analysis reveals, for the first time, a systematic performance gap of 8.8–18.4 percentage points for mainstream agents on non-English tasks, with 55% attributable to inherent model deficiencies and only 6.4% to translation errors. Further investigation identifies weak tool orchestration and the loss of morphological cues—exacerbated by ambiguity in non-Latin scripts—as key failure mechanisms. Notably, this performance disparity persists regardless of model scale.

Abstract

Agentic benchmarks aim to measure how well AI agents plan, search, execute, and recover within realistic multi-tool environments, but they are almost exclusively in English. As AI agents are globally deployed to a linguistically diverse user base, whether agentic competence measured in English transfers to other languages remains an open question. We introduce OmnilingualGAIA2, a machine-translated expansion (with partial human- expert validation) of the GAIA2 agentic benchmark, covering ten target languages spanning five writing systems, paired with a localised and human-calibrated multilingual verifier. Evaluating seven frontier and open-weight agents, we find a universal cross-lingual gap of 8.8-18.4 pass@3 points that is agent-asymmetric in magnitude, concentrates on tool-orchestration rather than quantitative reasoning, and does not close with model scale. A stratified error attribution decomposes the gap as predominantly model-driven (55%), with a bounded translation-contamination floor of only 6.4% of scenario-language pairs. Human-expert linguistic analysis further identifies morphological cue loss and amplified ambiguity as the primary failure mechanisms in non-Latin-script languages. Our results argue that multilingual agentic evaluation must become a standard part of the reporting protocol for globally deployed agents.

Problem

• multilingual gap
• AI agents
• agentic benchmark
• cross-lingual transfer
• language diversity

Innovation

• multilingual agentic benchmark
• cross-lingual gap
• tool orchestration
• error attribution
• non-Latin scripts