Fine-grained Testing for Autonomous Driving Software: a Study on Autoware with LLM-driven Unit Testing

2025-01-16

AI Summary

Low unit test coverage in the Autoware autonomous driving software and poor pass rates of large language model (LLM)-generated test cases hinder rigorous verification. To address this, we propose AwTest-LLM—the first LLM-augmented unit testing framework tailored for industrial-grade autonomous driving systems. AwTest-LLM integrates prompt engineering, code semantic analysis, and structural modeling of Autoware’s source code to devise architecture- and semantics-aware test generation and automated repair strategies. Experimental evaluation across multiple core Autoware packages demonstrates an average 37.2% improvement in test coverage, a 51.8% increase in test case pass rate, and the discovery of 12 previously uncovered critical boundary-condition logic defects. This work establishes the first source-code-level, fine-grained automated unit testing capability for Autoware and introduces a reusable, LLM-driven testing paradigm for safety-critical autonomous driving software validation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Testing autonomous driving systems (ADS) is critical to ensuring their reliability and safety. Existing ADS testing works focuses on designing scenarios to evaluate system-level behaviors, while fine-grained testing of ADS source code has received comparatively little attention. To address this gap, we present the first study on testing, specifically unit testing, for ADS source code. Our study focuses on an industrial ADS framework, Autoware. We analyze both human-written test cases and those generated by large language models (LLMs). Our findings reveal that human-written test cases in Autoware exhibit limited test coverage, and significant challenges remain in applying LLM-generated tests for Autoware unit testing. To overcome these challenges, we propose AwTest-LLM, a novel approach to enhance test coverage and improve test case pass rates across Autoware packages.

Problem

• Autonomous driving software
• Testing coverage
• Automated testing challenges

Innovation

• AwTest-LLM
• Autonomous Driving Systems
• LLM-enhanced Testing