NetPress: Dynamically Generated LLM Benchmarks for Network Applications

2025-06-03

AI Summary

Existing LLM agent evaluations rely heavily on static, small-scale benchmarks, failing to reflect real-world requirements—particularly in high-reliability domains such as network operations. Method: We propose the first dynamic, closed-loop evaluation framework tailored for network operations. It employs a unified state-action abstraction to support on-demand generation of million-scale, configurable queries and ground-truth labels. The framework deeply integrates containerized network emulators (e.g., EVE-NG, Colosseum) to jointly assess correctness, security compliance, and latency. Methodologically, it incorporates DSL-driven benchmark synthesis, automated ground-truth derivation, and multi-dimensional metrics. Contribution/Results: Evaluated across three canonical network tasks, our framework uncovers previously unobserved performance disparities among mainstream LLM agents—especially regarding security constraints, operational fault tolerance, and temporal sensitivity. These findings empirically validate the critical role of dynamic, closed-loop evaluation in assessing deployment readiness for production-grade network automation.

Technology Category

• Multiagent Systems: Agent-Based Simulation and Emergent Behavior
• Search and Optimization: Evaluation and Analysis
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Security and Privacy: Large-scale security measurements

Abstract

Despite growing interest in domain-specific benchmarking of large language models (LLMs) and agents, current evaluations remain limited to static, small-scale datasets, especially in high-stakes tasks like network operations that demand reliability for deployments. We present NetPress, an automated benchmark generation framework for evaluating LLM agents in network applications. NetPress introduces a unified abstraction with state and action, enabling dynamic generation of diverse query sets along with corresponding ground truths. At runtime, users can specify benchmark configurations to generate millions of queries on the fly. In addition to dynamic benchmark construction, NetPress integrates with network emulators to provide realistic environment feedback, supporting comprehensive evaluation across correctness, safety, and latency. We instantiate NetPress on three representative applications, revealing interesting fine-grained differences in agent behavior that static, correctness-only benchmarks often miss. NetPress moves LLM evaluation toward realistic, scalable testing in infrastructure-centric domains, helping close the gap between benchmark performance and real-world deployment readiness. Code is available at https://github.com/Froot-NetSys/NetPress.

Problem

• Dynamic benchmark generation for LLM evaluation in network applications
• Addressing limitations of static datasets in high-stakes network operations
• Integrating network emulators for realistic correctness, safety, and latency testing

Innovation

• Automated dynamic benchmark generation framework
• Unified abstraction with state and action
• Integration with network emulators for feedback