Automated Test Oracles for Flaky Cyber-Physical System Simulators: Approach and Evaluation

2025-08-28

AI Summary

In CPS simulation testing, simulator execution is time-consuming and unstable, leading to inconsistent results and necessitating repeated tests. To address this, this paper proposes an assertion-based oracle method that operates without system execution. Our approach innovatively integrates the spectrum-based fault localization metric Ochiai into a genetic programming framework to enhance oracle generation accuracy and robustness against simulator fluctuations. We also comparatively evaluate alternative models, including decision trees and decision rules. The method automatically constructs interpretable, logic- and arithmetic-based assertions over the input space. Evaluated on case studies from aerospace, networking, and autonomous driving domains, the Ochiai-enhanced genetic programming method achieves significantly higher accuracy than baseline approaches and demonstrates strong robustness to simulator variability, with an average accuracy fluctuation of only 4%.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data generation and simulation
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Simulation-based testing of cyber-physical systems (CPS) is costly due to the time-consuming execution of CPS simulators. In addition, CPS simulators may be flaky, leading to inconsistent test outcomes and requiring repeated test re-execution for reliable test verdicts. Automated test oracles that do not require system execution are therefore crucial for reducing testing costs. Ideally, such test oracles should be interpretable to facilitate human understanding of test verdicts, and they must be robust against the potential flakiness of CPS simulators. In this article, we propose assertion-based test oracles for CPS as sets of logical and arithmetic predicates defined over the inputs of the system under test. Given a test input, our assertion-based test oracle determines, without requiring test execution, whether the test passes, fails, or if the oracle is inconclusive in predicting a verdict. We describe two methods for generating assertion-based test oracles: one using genetic programming~(GP) that employs well-known spectrum-based fault localization (SBFL) ranking formulas, namely Ochiai, Tarantula, and Naish, as fitness functions; and the other using decision trees (DT) and decision rules (DR). We evaluate our assertion-based test oracles through case studies in the domains of aerospace, networking and autonomous driving. We show that test oracles generated using GP with Ochiai are significantly more accurate than those obtained using GP with Tarantula and Naish or using DT or DR. Moreover, this accuracy advantage remains even when accounting for the flakiness of the system under test. We further show that the assertion-based test oracles generated by GP with Ochiai are robust against flakiness with only 4% average variation in their accuracy results across four different network and autonomous driving systems with flaky behaviours.

Problem

• Reducing testing costs for flaky cyber-physical system simulators
• Creating interpretable automated test oracles without execution
• Ensuring robustness against simulator flakiness in test verdicts

Innovation

• Assertion-based oracles using logical predicates
• Genetic programming with Ochiai fitness function
• Robust oracles without requiring test execution