Test Adequacy for Metamorphic Testing: Criteria, Measurement, and Implication

2024-12-30

AI Summary

To address the lack of reliable and rigorous adequacy criteria for metamorphic testing (MT), this paper establishes, for the first time, an attribute-driven theoretical framework for MT adequacy grounded in the **essential properties** of the system under test. We propose a **computable adequacy metric** that jointly considers metamorphic relation (MR) coverage and the diversity/representativeness of source inputs, enabling objective, quantitative assessment of test quality. Unlike conventional approaches that focus solely on MR coverage or input count, our method holistically integrates semantic correctness with input quality. Empirical evaluation demonstrates that the proposed metric exhibits a strong positive correlation with fault-detection capability: test suites achieving higher adequacy yield an average 23.6% improvement in fault detection rate. Moreover, the metric effectively guides test suite optimization.

Technology Category

• Search and Optimization: Metareasoning and Metaheuristics
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Metamorphic testing (MT) is a simple yet effective technique to alleviate the oracle problem in software testing. The underlying idea of MT is to test a software system by checking whether metamorphic relations (MRs) hold among multiple test inputs (including source and follow-up inputs) and the actual output of their executions. Since MRs and source inputs are two essential components of MT, considerable efforts have been made to examine the systematic identification of MRs and the effective generation of source inputs, which has greatly enriched the fundamental theory of MT since its invention. However, few studies have investigated the test adequacy assessment issue of MT, which hinders the objective measurement of MT's test quality as well as the effective construction of test suites. Although in the context of traditional software testing, there exist a number of test adequacy criteria that specify testing requirements to constitute an adequate test from various perspectives, they are not in line with MT's focus which is to test the software under testing (SUT) from the perspective of necessary properties. In this paper, we proposed a new set of criteria that specifies testing requirements from the perspective of necessary properties satisfied by the SUT, and designed a test adequacy measurement that evaluates the degree of adequacy based on both MRs and source inputs. The experimental results have shown that the proposed measurement can effectively indicate the fault detection effectiveness of test suites, i.e., test suites with increased test adequacy usually exhibit higher effectiveness in fault detection. Our work made an attempt to assess the test adequacy of MT from a new perspective, and our criteria and measurement provide a new approach to evaluate the test quality of MT and provide guidelines for constructing effective test suites of MT.

Problem

• Mutation Testing
• Test Quality
• Effectiveness Measurement

Innovation

• Mutation Testing
• Evaluation Metrics
• Software Fault Detection