Continuous Benchmark Generation for Evaluating Enterprise-scale LLM Agents

2025-11-13

AI Summary

Enterprise-scale LLM agents face persistent evaluation challenges due to dynamic service evolution and scarcity of realistic, annotated test cases. Method: This paper proposes a dynamic benchmark generation approach grounded in semi-structured documentation, leveraging intent extraction and LLM-driven test case synthesis to automatically construct maintainable, business-aligned evaluation benchmarks that evolve with operational requirements—without reliance on dense human annotation and robust even under sparse ground-truth conditions. Contribution/Results: Compared to static benchmarks, our method significantly reduces maintenance overhead while improving coverage and responsiveness. Empirical validation in large-scale enterprise service migration scenarios demonstrates a 3.2× improvement in benchmark construction efficiency, enabling rapid agent iteration and closed-loop feedback for continuous optimization.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The rapid adoption of AI agents across domains has made systematic evaluation crucial for ensuring their usefulness and successful production deployment. Evaluation of AI agents typically involves using a fixed set of benchmarks and computing multiple evaluation metrics for the agent. While sufficient for simple coding tasks, these benchmarks fall short for enterprise-scale agents, where services and requirements evolve continuously and ground-truth examples are sparse. We propose a process of benchmark generation that helps evolve the benchmarks as the requirements change and perform robust evaluation of evolving AI agents. We instantiate this approach for a case study of service migration from one deployment platform to another at a large public enterprise. Our approach relies on semi-structured documents where developers express the high-level intent, and uses state-of-the-art LLMs to generate benchmarks from just a small number of such documents. Overall, this process results in a maintainable evaluation framework, enabling rapid feedback on agent performance and facilitating targeted improvements.

Problem

• Evaluating evolving enterprise-scale AI agents with sparse ground-truth data
• Generating continuous benchmarks to match changing service requirements
• Creating maintainable evaluation frameworks using minimal semi-structured documents

Innovation

• Generates benchmarks from semi-structured developer documents
• Uses state-of-the-art LLMs for benchmark creation
• Creates maintainable framework for evolving agent evaluation