Telco-GAIA: Bilingual Benchmark for Agents in Telecom Domain

2026-07-04

AI Summary

该研究通过构建Telco-GAIA基准，使用多模态数据评估电信领域内代理工具的应用能力，涵盖英语和阿拉伯语的复杂查询任务。

Abstract

We introduce Telco-GAIA, a bilingual, multi-modal benchmark for evaluating tool-using agents on the data of a real-world telecommunications operator. Telco-GAIA comprises 100 human-verified question-answering tasks, in English and Arabic, that each demand multi-hop reasoning (4.2 hops on average) over three heterogeneous sources: a static website snapshot (HTML, images, and linked PDFs), a synthetic relational SQL database, and external web archives, spanning text, image, and tabular modalities. The benchmark is delivered as a sandboxed Docker environment and scored by normalized exact string matching, making evaluation objective, deterministic, and reproducible over time without any LLM-as-a-Judge. Evaluating a purpose-built reference agent across twelve commercial and open LLMs, we find Telco-GAIA challenging: even the strongest model solves only 71% of tasks; under a moderate cost budget, this falls to about 40%, and the visually grounded categories remain the weakest, where the average backend scores below 30%, leaving substantial headroom in document and image understanding. Telco-GAIA offers a rigorous, reproducible testbed for enterprise agents and a template for constructing closed-domain benchmarks.

Problem

• bilingual benchmark
• multi-modal
• telecommunications

Innovation

• bilingual benchmark
• multi-modal reasoning
• telecommunications domain
• reproducible evaluation
• heterogeneous data sources