🤖 AI Summary
Large language models (LLMs) deployed for industrial test generation face critical reliability challenges due to rapid model iteration, leading to outdated evaluations and compromised production trustworthiness.
Method: This paper introduces the first continuous evaluation framework for LLM-based test generation tailored to industrial settings. It pioneers a “continuous evaluation” paradigm integrating technical metrics (e.g., code coverage) with engineering metrics (e.g., maintainability, expert ratings), while systematically addressing real-world issues including data leakage and irreproducible results. The framework integrates industrial toolchains (e.g., SonarQube), supports dynamic test-case selection, robust prompt engineering, and auditable measurement infrastructure.
Contribution/Results: A longitudinal empirical study at LKS Next demonstrates that the framework accurately tracks LLM capability evolution, identifies key bottlenecks impeding industrial deployment, and effectively enables trustworthy integration into DevSecOps pipelines.
📝 Abstract
Large Language Models (LLMs) have shown great potential in automating software testing tasks, including test generation. However, their rapid evolution poses a critical challenge for companies implementing DevSecOps - evaluations of their effectiveness quickly become outdated, making it difficult to assess their reliability for production use. While academic research has extensively studied LLM-based test generation, evaluations typically provide point-in-time analyses using academic benchmarks. Such evaluations do not address the practical needs of companies who must continuously assess tool reliability and integration with existing development practices. This work presents a measurement framework for the continuous evaluation of commercial LLM test generators in industrial environments. We demonstrate its effectiveness through a longitudinal study at LKS Next. The framework integrates with industry-standard tools like SonarQube and provides metrics that evaluate both technical adequacy (e.g., test coverage) and practical considerations (e.g., maintainability or expert assessment). Our methodology incorporates strategies for test case selection, prompt engineering, and measurement infrastructure, addressing challenges such as data leakage and reproducibility. Results highlight both the rapid evolution of LLM capabilities and critical factors for successful industrial adoption, offering practical guidance for companies seeking to integrate these technologies into their development pipelines.