ICST Tool Competition 2025 - Self-Driving Car Testing Track

2025-02-14

AI Summary

Low efficiency and inconsistent evaluation in test case selection for autonomous vehicle (AV) simulation-based regression testing hinder progress in this domain. Method: This paper introduces the first dedicated tool competition platform for AV simulation testing, featuring a standardized test environment, a curated dataset of realistic driving scenarios, and a multidimensional evaluation benchmark—encompassing coverage, fault detection rate, and regression sensitivity. It pioneers a formal tool competition framework and proposes a quantitative evaluation methodology integrating baseline comparisons. Contribution/Results: Five independently developed test selection tools were systematically evaluated on the platform, revealing substantial performance disparities across algorithms in both testing efficiency and fault detection capability. This work establishes the field’s first reproducible, extensible benchmark for assessing AV simulation test tools, providing critical infrastructure and methodological foundations for future research in AV verification and validation.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Constraint Satisfaction and Optimization: Solvers and Tools

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Security and Privacy: Large-scale security measurements

Abstract

This is the first edition of the tool competition on testing self-driving cars (SDCs) at the International Conference on Software Testing, Verification and Validation (ICST). The aim is to provide a platform for software testers to submit their tools addressing the test selection problem for simulation-based testing of SDCs, which is considered an emerging and vital domain. The competition provides an advanced software platform and representative case studies to ease participants' entry into SDC regression testing, enabling them to develop their initial test generation tools for SDCS. In this first edition, the competition includes five tools from different authors. All tools were evaluated using (regression) metrics for test selection as well as compared with a baseline approache. This paper provides an overview of the competition, detailing its context, framework, participating tools, evaluation methodology, and key findings.

Problem

• Addresses test selection for SDC simulation testing.
• Provides platform for SDC regression testing tools.
• Evaluates tools using regression metrics and baseline.

Innovation

• Simulation-based testing for SDCs
• Advanced software platform provided
• Regression metrics for evaluation