Simulink Mutation Testing using CodeBERT

2025-01-13

AI Summary

This work addresses the poor generalizability and limited adaptability of manually defined mutation operators in Simulink model mutation testing—particularly for requirement-driven scenarios. We introduce CodeBERT, a pre-trained code language model, to Simulink block-level mutant generation for the first time. Our approach is purely data-driven and text-based: it converts Simulink model structures into learnable token sequences and leverages masked language modeling to automatically discover semantically meaningful and syntactically valid mutation patterns—eliminating reliance on handcrafted rules. Evaluated on an industrial-scale Simulink benchmark, our method fully reproduces all known block-level mutation patterns from the literature and complements the state-of-the-art tool FIM. Under requirement-aware evaluation, our mutation operators demonstrate significantly higher effectiveness and fault-detection capability than FIM, validating a novel large-model-driven paradigm for mutation generation.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Web Mining and Content Analysis: Web data generation and simulation
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We present BERTiMuS, an approach that uses CodeBERT to generate mutants for Simulink models. BERTiMuS converts Simulink models into textual representations, masks tokens from the derived text, and uses CodeBERT to predict the masked tokens. Simulink mutants are obtained by replacing the masked tokens with predictions from CodeBERT. We evaluate BERTiMuS using Simulink models from an industrial benchmark, and compare it with FIM -- a state-of-the-art mutation tool for Simulink. We show that, relying exclusively on CodeBERT, BERTiMuS can generate the block-based Simulink mutation patterns documented in the literature. Further, our results indicate that: (a) BERTiMuS is complementary to FIM, and (b) when one considers a requirements-aware notion of mutation testing, BERTiMuS outperforms FIM.

Problem

• CodeBERT
• Simulink Model Errors
• Performance Comparison

Innovation

• BERTiMuS
• CodeBERT
• Simulink Model Transformation