Benchmarking LLM-Driven Network Configuration Repair

2026-04-24

AI Summary

This work addresses the lack of effective benchmarks for evaluating large language models’ (LLMs) ability to repair errors in large-scale, interdependent network configurations without introducing new faults. The authors propose Cornetto, the first LLM-based network repair benchmark that enables scalable and functionally correct verification. Cornetto employs a configuration scenario generation pipeline to synthesize diverse, realistic misconfigurations and integrates formal verification with multi-protocol topology modeling to rigorously assess repair correctness. The benchmark comprises 231 problems spanning networks of 20 to 754 nodes. Evaluations of nine state-of-the-art LLMs reveal a significant performance degradation as network scale increases, along with a pronounced tendency to introduce regression errors during repair attempts.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

There is a rapidly growing interest in using Large Language Models (LLMs) to automate complex network operations, but their reliable adoption requires rigorous assessment of their effectiveness and safety. Existing benchmarks do not address whether LLMs can successfully resolve errors in large-scale, interdependent network configurations without introducing new disruptions. Developing such a benchmark is challenging: scenarios must be diverse and increasingly complex, yet their evaluation must be straightforward and meaningful. In this paper, we present Cornetto, the first benchmark to evaluate LLM-driven network configuration repair functionally and at scale. Cornetto features a generation pipeline that synthesizes representative and plausible misconfiguration scenarios, coupled with an evaluation framework that uses formal verification to assess functional correctness of proposed fixes against ground-truth specifications. Using this pipeline, we synthesize a dataset of 231 problems for fixing configurations across varying network topologies (20--754 nodes) and diverse protocols. We evaluate 9 state-of-the-art LLMs and find that while they show promise, they often introduce regressions and their performance degrades at scale. Our results indicate that reliable LLM-powered network automation requires integrating LLMs into iterative workflows guided by formal verification.

Problem

• LLM
• network configuration repair
• benchmarking
• misconfiguration
• formal verification

Innovation

• Large Language Models
• Network Configuration Repair
• Formal Verification
• Benchmarking
• Misconfiguration Synthesis