Testing CPS with Design Assumptions-Based Metamorphic Relations and Genetic Programming

2024-12-04

IEEE Transactions on Software Engineering

AI Summary

In cyber-physical system (CPS) testing, the oracle problem arises due to restricted input trajectories, hindering validation of system behavior. Method: This paper proposes a design-assumption-driven metamorphic testing approach: it constructs metamorphic relations (MRs) grounded in control-theoretic design assumptions—rather than conventional functional requirements—and employs genetic programming to automatically synthesize arbitrary-shaped input trajectories along with their expected output trajectories; MR violations are then identified via deviation detection. Contribution/Results: The method eliminates reliance on simplistic stimuli (e.g., step or ramp inputs), enabling failure-oriented, directed test generation. Experiments demonstrate a substantial increase in MR violation detection rate, provide interpretable pass/fail criteria rooted in design intent, and effectively handle the non-triviality of the search space.

Technology Category

• Search and Optimization: Metareasoning and Metaheuristics
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data generation and simulation
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Cyber-Physical Systems (CPSs) software is used to enforce desired behaviours on physical systems. To test the interaction between the CPS software and the system's physics, engineers provide traces of desired physical states and observe traces of the actual physical states. CPS requirements describe how closely the actual physical traces should track the desired traces. These requirements are typically defined for specific, simple input traces such as step or ramp sequences, and thus are not applicable to arbitrary inputs. This limits the availability of oracles for CPSs. Our recent work proposes an approach to testing CPS using control-theoretical design assumptions instead of requirements. This approach circumvents the oracle problem by leveraging the control-theoretical guarantees that are provided when the design assumptions are satisfied. To address the test case generation and oracle problems, researchers have proposed metamorphic testing, which is based on the study of relations across tests, i.e., metamorphic relations (MRs). In this work, we define MRs based on the design assumptions and explore combinations of these MRs using genetic programming to generate CPS test cases. This enables the generation of CPS input traces with potentially arbitrary shapes, together with associated expected output traces. We use the deviation from the expected output traces to guide the generation of input traces that falsify the MRs. Our experiment results show that the MR-falsification provides engineers with new information, helping them identify passed and failed test cases. Furthermore, we show that the generation of traces that falsify the MRs is a non-trivial problem, which is successfully addressed by our genetic search.

Problem

• Testing CPS software using design assumptions instead of requirements
• Generating arbitrary input traces with expected outputs via metamorphic relations
• Falsifying metamorphic relations to identify passed and failed test cases

Innovation

• Uses design assumptions for metamorphic relations
• Applies genetic programming for test generation
• Generates arbitrary input-output trace pairs