PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents

2026-07-07

AI Summary

Current evaluations of large language model agents are predominantly confined to monolingual settings, failing to capture the complexity of real-world multilingual workflows. This work proposes PolyWorkBench—the first benchmark designed for evaluating agents on multilingual, long-horizon tasks—encompassing 67 tasks across five domains that require agents to process heterogeneous multilingual inputs, perform iterative reasoning, invoke external tools, and produce structured outputs. We introduce a hybrid automated evaluation framework that integrates structural matching, program executability verification, and semantic scoring via large language models. Experimental results demonstrate a significant performance degradation of state-of-the-art agents in multilingual scenarios, revealing that linguistic diversity imposes compounded negative effects on both reasoning and execution stages.

Abstract

Large language model (LLM) agents have shown strong performance in long-horizon tasks that require planning, tool use, and interaction with external environments. However, most existing benchmarks implicitly assume a monolingual setting, where the entire execution process, including reasoning, tool invocation, and output generation, is conducted within a single language. In contrast, real-world applications often involve multilingual inputs and outputs within a unified workflow, yet the interaction between multilinguality and agentic execution remains underexplored. In this work, we introduce PolyWorkBench, a benchmark for evaluating LLM agents on multilingual long-horizon workplace workflows. PolyWorkBench consists of 67 tasks across five domains, including commerce, knowledge work, legal analysis, localization, and manufacturing, where agents must process heterogeneous multilingual inputs, perform iterative reasoning, invoke external tools, and produce structured outputs. To enable comprehensive evaluation, we propose a hybrid framework that combines structural grading, executable verification, and LLM-based semantic assessment. This design allows us to capture both functional correctness and linguistic consistency across complex workflows. Empirical results show that state-of-the-art LLM agents suffer significant performance degradation in multilingual workflow settings compared to monolingual counterparts. Our analysis suggests that multilinguality introduces compounding effects across reasoning and execution steps, highlighting the importance of jointly modeling language variation and procedural decision-making in agent evaluation.

Problem

• multilingual
• LLM agents
• long-horizon tasks
• benchmarking
• workplace workflows

Innovation

• multilingual LLM agents
• long-horizon tasks
• benchmarking
• workflow evaluation
• hybrid assessment framework