Using Large Language Models for Black-Box Testing of FMU-Based Simulations

2026-04-28

AI Summary

This work proposes a novel approach to black-box testing of Functional Mock-up Units (FMUs) by integrating large language models (LLMs) with a human-in-the-loop mechanism. Addressing the inefficiency and poor interpretability of traditional FMU-based dynamic simulation testing—which relies on manually crafted scenarios—the method automatically generates structured Given-When-Then test objectives from FMU interface and functional specifications, and constructs complete test plans comprising input sequences and assertion oracles. Upon simulation execution, the framework produces visualizable logs and statistical evaluation metrics. The approach significantly enhances test design efficiency and result interpretability, facilitates test asset reuse, and demonstrates effectiveness on a lubricating oil cooling system by autonomously generating executable test scenarios and delivering objective-level pass-rate analysis.

Abstract

We propose a human in the loop approach for black-box testing of Functional Mock-up Units (FMUs) using Large Language Models (LLMs). The goal is to reduce the manual effort in defining test scenarios for dynamic simulation models and to improve the interpretability of results. The approach takes the functional and interface specifications of an FMU as input, and prompts an LLM to generate structured scenario goals in Given-When-Then format that define the initial input conditions of the simulation, a possible change in those conditions, and the expected output behaviour of the system against those changes. The corresponding scenario plans specify input patterns and add assertion oracles that describe expected output patterns defined in scenario goals. The approach generates a complete input time series for the scenario plans, runs the FMU simulation, and evaluates assertions on the recorded outputs. It produces human-readable logs and plots that show statistics for each scenario with overlays, aggregate pass rates, and per-goal outcomes. The generated scenarios and results are stored for evaluation and later re-execution. We evaluate the approach on a Lube Oil Cooling system and discuss design choices that make the approach practical for everyday use. Results suggest that LLM-assisted scenario generation can facilitate automatic test design and verification of dynamic simulation models.

Problem

• black-box testing
• Functional Mock-up Unit
• Large Language Models
• test scenario generation
• dynamic simulation

Innovation

• Large Language Models
• Black-Box Testing
• Functional Mock-up Units
• Scenario Generation
• Assertion Oracles