Test Case Generation for Simulink Models: An Experience from the E-Bike Domain

2025-01-10

AI Summary

This study investigates the effectiveness and efficiency of Search-Based Software Testing (SBST) in uncovering design defects within a Simulink model of an electric bicycle (e-Bike) motor controller—a complex industrial embedded system. We propose and, for the first time, systematically evaluate the HECATE framework on a real-world e-Bike industrial controller; it integrates coverage-driven optimization with a multi-objective fitness function to jointly validate functional, safety, and regulatory requirements. Across 36 experiments, SBST identified 30 developer-confirmed failures (83% detection rate), with an average execution time of 1 hour 17 minutes. Results strongly demonstrate SBST’s industrial applicability to embedded control software. Key contributions are: (1) the first empirically grounded SBST benchmark specifically for the e-Bike domain; and (2) the successful transition of the HECATE framework from academic prototype to industrial practice.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Constraint Satisfaction and Optimization: Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Cyber-physical systems development often requires engineers to search for defects in their Simulink models. Search-based software testing (SBST) is a standard technology that supports this activity. To increase practical adaption, industries need empirical evidence of the effectiveness and efficiency of (existing) SBST techniques on benchmarks from different domains and of varying complexity. To address this industrial need, this paper presents our experience assessing the effectiveness and efficiency of SBST in generating failure-revealing test cases for cyber-physical systems requirements. Our study subject is within the electric bike (e-Bike) domain and concerns the software controller of an e-Bike motor, particularly its functional, regulatory, and safety requirements. We assessed the effectiveness and efficiency of HECATE, an SBST framework for Simulink models, to analyze two software controllers. HECATE successfully identified failure-revealing test cases for 83% (30 out of 36) of our experiments. It required, on average, 1 h 17 min 26 s (min = 11 min 56 s, max = 8 h 16 min 22 s, std = 1 h 50 min 34 s) to compute the failure-revealing test cases. The developer of the e-Bike model confirmed the failures identified by HECATE. We present the lessons learned and discuss the relevance of our results for industrial applications, the state of practice improvement, and the results' generalizability.

Problem

• Search-Based Software Testing
• Simulink Models
• Error Detection

Innovation

• Search-Based Software Testing
• Electric Bicycle Motor Controller
• HECATE Tool