NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration

2026-08-24

AI Summary

为解决LLM代理在网络配置中的可靠性问题，本文提出NetConfArena基准测试，通过在模拟多设备网络中评估LLM代理的行为来改进其性能。

Abstract

Large language model (LLM) agents are increasingly attractive for automating network configuration, yet their reliability and failure patterns are poorly understood. An essential prerequisite is to assess such agents in a realistic but risk-free environment. Existing benchmarks, however, fall short: they often treat configuration as static command generation or rely on overly simplified settings. Such evaluations understate the core challenges of network configuration, where correctness requires reasoning about protocol complexity and topology dependence. We present NetConfArena, an executable benchmark for evaluating LLM agents in closed-loop network configuration. NetConfArena places agents in emulated multi-device networks, provides a standardized and compact action interface for task execution, and evaluates the resulting network behavior with hidden task-specific executable test cases. The benchmark relies on an LLM-assisted, emulation-grounded pipeline, which converts human-oriented network materials into reusable parameterized task templates. We evaluate representative LLM agents on 480 task instances instantiated from 96 protocol-focused task templates, yielding 3840 execution trajectories, and show that failures are not limited to command errors. The failures also reveal gaps in task-specification adherence and robust planning and execution. These findings suggest two future directions: using validated trajectories as supervision signals to improve foundation models, and designing harness mechanisms that make agent execution more reliable and accountable.

Problem

• large language model
• network configuration
• benchmark
• reliability
• failure patterns

Innovation

• executable benchmark
• closed-loop network configuration
• emulated multi-device networks
• parameterized task templates
• LLM-assisted pipeline