Intent-Based Mutation Testing: From Naturally Written Programming Intents to Mutants

2026-07-06

AI Summary

Traditional mutation testing, which operates at the syntactic level, struggles to detect semantic defects arising from misunderstandings of program intent. This work proposes an intent-based mutation testing approach that, for the first time, treats programming intent itself as the unit of mutation. Leveraging large language models (LLMs), the method semantically rewrites natural language descriptions of intent and automatically generates executable program mutants. By shifting the focus from syntactic transformations to semantic reinterpretations of intent, this approach produces mutants that are both richer in semantics and more structurally complex. Empirical evaluation on 29 programs shows that 55% of the intent-based mutants are not covered by traditional mutation operators and exhibit significant differences from them in both syntactic and semantic dimensions, thereby substantially enhancing the detection of specification- and behavior-level faults.

Abstract

This paper presents intent-based mutation testing, a testing approach that generates mutations by changing the programming intents that are implemented in the programs under test. In contrast to traditional mutation testing, which changes (mutates) the way programs are written, intent mutation changes (mutates) the behavior of the programs by producing mutations that implement (slightly) different intents than those implemented in the original program. The mutations of the programming intents represent possible corner cases and misunderstandings of the program behavior, i.e., program specifications, and thus can capture different classes of faults than traditional (syntax-based) mutation. Moreover, since programming intents can be implemented in different ways, intent-based mutation testing can generate diverse and complex mutations that are close to the original programming intents (specifications) and thus direct testing towards the intent variants of the program behavior/specifications. We implement intent-based mutation testing using Large Language Models (LLMs) that mutate programming intents and transform them into mutants. We evaluate intent-based mutation on 29 programs and show that it generates mutations that are syntactically complex, semantically diverse, and quite different (semantically) from the traditional ones. We also show that 55% of the intent-based mutations are not subsumed by traditional mutations. Overall, our analysis shows that intent-based mutation testing can be a powerful complement to traditional (syntax-based) mutation testing.

Problem

• mutation testing
• programming intent
• program specification
• semantic diversity
• fault detection

Innovation

• intent-based mutation testing
• Large Language Models
• semantic mutation
• program specification
• mutation testing