CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations

2026-08-12

AI Summary

Existing evaluation methodologies struggle to emulate the multi-vendor, multi-protocol, and partially observable environments characteristic of real-world telecommunications networks, thereby failing to effectively assess AI agents’ capabilities in fault diagnosis and path restoration. To address this gap, this work proposes CTBench—the first public benchmark tailored for telecom operations—focusing on root cause analysis and path restoration tasks. Developed with expert-crafted simulation scenarios and annotated gold-standard evidence chains, CTBench introduces an expert-aligned evaluation metric that accounts for both final answers and diagnostic reasoning, emphasizing interpretability and evidential support, while also providing fine-grained metadata. Experimental results demonstrate that state-of-the-art agents perform reasonably well in path restoration but exhibit significant deficiencies in complex root cause analysis—particularly involving interface states, link-layer issues, and service management failures—and often lack sufficient diagnostic evidence to substantiate their conclusions.

Abstract

Agents are increasingly considered for automating network operations and maintenance, where engineers must diagnose network faults, optimize configurations to enhance services, and reduce operational costs while acting under strict constraints. However, existing evaluations fail to accurately model real network characteristics or assess agents under partially observable telecom environments with diverse vendors, devices, protocols, and interfaces. In this paper, we introduce CTBench, a public benchmark for assessing whether an agent behaves like a competent telecom troubleshooting engineer. CTBench focuses on root cause analysis and path restoration. Each task is constructed by experts and annotated with rich task metadata, including golden evidence steps. CTBench uses expert-grounded metrics that evaluate both final answers and the diagnostic evidence. Experiments with representative harness-model combinations show that state-of-the-art agents perform very well at identifying endpoints in path-restoration tasks but, more generally, underperform in root cause analysis. In particular, agents struggle with interface state, link-layer, service-management, and other operational faults. Most importantly, even when agents produce plausible or correct final answers, they often fail to provide the evidence-grounded diagnoses required in operational practice. Our results further show that path restoration is generally more resource expensive, yet larger resource usage does not necessarily translate into better diagnosis.

Problem

• troubleshooting
• telecom network operations
• root cause analysis
• path restoration
• AI agent evaluation

Innovation

• CTBench
• telecom troubleshooting
• root cause analysis
• evidence-grounded diagnosis
• AI agent evaluation