SEATauBench: Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages

2026-06-26

AI Summary

This work addresses the inadequacy of existing evaluation frameworks in effectively assessing agent capabilities for low-resource languages in Southeast Asia, which fails to reflect the real-world performance of sovereign AI in local contexts. To bridge this gap, we propose SEATauBench—the first multilingual agent benchmark tailored for Southeast Asian sovereign AI—extending the tool-agent-user evaluation paradigm to this region and introducing a reusable, multi-tiered localization pipeline encompassing conversational language, tool descriptions, and task domains. Cross-lingual transfer experiments based on TauBench reveal that while model performance remains relatively stable when only the conversational language is switched, it degrades substantially as localization depth increases, particularly under full-domain adaptation. These findings underscore the severe limitations of evaluations relying solely on English-centric benchmarks.

Abstract

While AI development and evaluation for Southeast Asia (SEA) has grown rapidly, agent capabilities in regional languages are still poorly understood despite its importance to sovereign AI. To fill this gap, we introduce SEATauBench, the first agent-focused evaluation framework for SEA sovereign AI. SeaTau adapts TauBench to five languages -- Mandarin, Vietnamese, Thai, Indonesian, and Filipino -- and evaluates agents across progressively localized settings that vary the language of user-agent interaction, tool specifications, and task domains. Across three recent models, we find that English agent capabilities transfer reasonably well when only the conversation language changes, but quality and robustness degrade sharply as more task contexts are localized, with the largest losses in full domain adaptation. We also the limits of English-only agent assessment for measuring agent capabilities in SEA languages. More broadly, SeaTau provides a diagnostic benchmark and reusable adaptation pipeline for building reliable multilingual agents for linguistically diverse regions. Data and code can be accessed at github.com/SEACrowd/SEATauBench.

Problem

• Southeast Asian languages
• agent evaluation
• low-resource languages
• sovereign AI
• multilingual agents

Innovation

• multilingual agent evaluation
• low-resource languages
• sovereign AI
• localization robustness
• SEATauBench