LLM-Based Config Synthesis requires Disambiguation

2025-07-16

AI Summary

In LLM-driven incremental synthesis of network configurations (e.g., route maps and ACLs), ambiguous user intents frequently cause overlapping rule headers, hindering autonomous priority resolution—empirical analysis reveals hundreds of ACL overlaps in cloud environments. To address this, we propose Clarify, a novel system featuring a lightweight Disambiguator module that guides users interactively to resolve intent ambiguities and integrates formal logic verification for verifiable incremental configuration synthesis. Unlike end-to-end black-box generation, Clarify explicitly models intent disambiguation as a collaborative reasoning step, ensuring both security guarantees and policy accuracy. Evaluation demonstrates that Clarify significantly reduces configuration conflict rates and, for the first time, systematically identifies and resolves LLM synthesis ambiguities arising from rule overlap in real-world network deployments.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Algorithm Configuration

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Beyond hallucinations, another problem in program synthesis using LLMs is ambiguity in user intent. We illustrate the ambiguity problem in a networking context for LLM-based incremental configuration synthesis of route-maps and ACLs. These structures frequently overlap in header space, making the relative priority of actions impossible for the LLM to infer without user interaction. Measurements in a large cloud identify complex ACLs with 100's of overlaps, showing ambiguity is a real problem. We propose a prototype system, Clarify, which uses an LLM augmented with a new module called a Disambiguator that helps elicit user intent. On a small synthetic workload, Clarify incrementally synthesizes routing policies after disambiguation and then verifies them. Our treatment of ambiguities is useful more generally when the intent of updates can be correctly synthesized by LLMs, but their integration is ambiguous and can lead to different global behaviors.

Problem

• LLMs struggle with user intent ambiguity in config synthesis
• Route-maps and ACLs overlap, causing priority inference issues
• Ambiguity in updates leads to divergent global behaviors

Innovation

• LLM augmented with Disambiguator module
• Incremental synthesis of routing policies
• Verification after disambiguation