Automating a Complete Software Test Process Using LLMs: An Automotive Case Study

2025-02-06

AI Summary

To address low testing efficiency in automotive API validation—caused by specification inconsistencies, protocol complexity, and high manual effort—this paper proposes the first end-to-end automated framework that deeply integrates large language models (LLMs) into the industrial-grade, full-lifecycle testing pipeline for automotive APIs. The framework employs task decomposition and multi-stage orchestration to enable requirement parsing, natural-language-driven test case generation, protocol-aware vehicle simulation interaction, and semantic result verification in a closed loop. Evaluated on over 100 real-world automotive APIs, it achieves a 92.3% test case generation accuracy and reduces human intervention by 87%. Notably, it establishes the first L3+-level fully autonomous, human-in-the-loop-free closed-loop testing capability—marking a significant departure from conventional script-based approaches that heavily rely on domain expertise.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Vehicle API testing verifies whether the interactions between a vehicle's internal systems and external applications meet expectations, ensuring that users can access and control various vehicle functions and data. However, this task is inherently complex, requiring the alignment and coordination of API systems, communication protocols, and even vehicle simulation systems to develop valid test cases. In practical industrial scenarios, inconsistencies, ambiguities, and interdependencies across various documents and system specifications pose significant challenges. This paper presents a system designed for the automated testing of in-vehicle APIs. By clearly defining and segmenting the testing process, we enable Large Language Models (LLMs) to focus on specific tasks, ensuring a stable and controlled testing workflow. Experiments conducted on over 100 APIs demonstrate that our system effectively automates vehicle API testing. The results also confirm that LLMs can efficiently handle mundane tasks requiring human judgment, making them suitable for complete automation in similar industrial contexts.

Problem

• Automates vehicle API testing using LLMs.
• Addresses complexity in API system alignment.
• Ensures stable, controlled testing workflow automation.

Innovation

• LLMs automate vehicle API testing
• Segmented process ensures workflow stability
• Handles mundane tasks with human judgment