ModBench: A Pipeline for Building Modelica Benchmark Datasets Mined from Library Repositories

2026-08-17

AI Summary

This study addresses the absence of benchmark datasets for Modelica, which has hindered empirical research on model evolution. We propose ModBench, an automated pipeline that establishes a novel paradigm for generating model snapshot benchmarks directly from code repositories by mining Git history, filtering commits, extracting simulatable classes, and normalizing representations. Applying this approach to the Modelica Standard Library, we constructed a comprehensive dataset comprising 85,562 class snapshots spanning all versions since v3, complete with API access and traceability links. This work fills a critical data gap in the domain, providing essential infrastructure to support research in model evolution analysis, compiler testing, and automated program repair.

Abstract

Research on equation-based cyber-physical systems modeling languages, such as Modelica, is constrained by the lack of curated benchmark datasets. This limits empirical insight into the evolution and development of models. We address this gap with ModBench, a pipeline that mines Git repositories of Modelica libraries to produce benchmark datasets of model snapshots. The pipeline (1) filters repository commits to retain human-authored, Modelica-relevant revisions; (2) extracts simulation-eligible classes; and (3) builds canonical representations of Modelica classes. For empirical validation, we applied ModBench to the Modelica Standard Library (MSL) and report the resulting dataset, spanning the full commit history (since Modelica language v3), with 85,562 distinct class snapshots, and links enabling traceability to original models and Git metadata. The dataset, its API, and the data generation pipeline are publicly available to support future research on model evolution analysis, compiler testing, and automated model repair or generation.

Problem

• Modelica
• benchmark datasets
• cyber-physical systems
• model evolution

Innovation

• ModBench
• Modelica
• Benchmark Dataset
• Repository Mining
• Cyber-Physical Systems