Breaking Models to Test the Judge: A Mutation Testing Approach for Semantic Evaluators of Domain Class Diagrams

2026-08-14

AI Summary

This study addresses the lack of benchmarks and high costs associated with manual validation in semantic modeling evaluation by proposing a mutation testing framework for domain class diagrams. By defining eleven semantic mutation operators to inject controllable defects, we establish an automated and scalable method for assessing evaluator detection capabilities. Experiments demonstrate that results from this automated mutation testing align closely with human evaluations across multiple LLM-based judge configurations. This work pioneers a mutation testing paradigm for class diagram semantic evaluation, effectively validating its reliability as a substitute for manual assessment and providing a standardized verification mechanism for semantic analysis agents.

Abstract

In software engineering, many semantic modeling tasks lack a unique ground truth, as human judgments are both costly and subjective. This paper explores mutation testing as a scalable alternative for evaluating semantic judges (e.g., LLM-based) of models. We propose a mutation testing approach in which controlled semantic defects are injected into domain class diagrams. Starting from pairs of PlantUML class diagrams and textual system descriptions, we apply mutation operators (e.g., removing a class) to generate faulty variants. A candidate judge is then evaluated based on its ability to detect the injected defects. We define 11 mutation operators for the task of comparing a domain class diagram against a textual description and evaluate the proposed approach against a conventional manual assessment of judgment validity. Across six judge configurations (three LLMs and two prompt variants), the automated mutation testing approach is largely consistent with the manual assessment in identifying the better-performing configurations. The results suggest that mutation testing may serve as a scalable proxy for analyzing semantic judges.

Problem

• Semantic Evaluators
• Mutation Testing
• Domain Class Diagrams
• Ground Truth
• Model Evaluation

Innovation

• Mutation Testing
• Semantic Evaluators
• Domain Class Diagrams
• Automated Assessment
• Large Language Models