Completeness of FSM Test Suites Reconsidered

2024-10-25

arXiv.org

AI Summary

Traditional fault-domain assumptions in FSM black-box conformance testing—requiring implementations to have only slightly more states than the specification—are unrealistic for industrial systems exhibiting exponential state-space redundancy. Method: We propose the practical *k-A-completeness* model, which constrains the *depth of reachability* of extra states rather than their count, enabling coverage of implementations with exponentially many redundant states. We formally define k-A-completeness and derive a sufficient condition based on test trees. Contribution/Results: We prove that the Wp and HSI test methods satisfy k-A-completeness, whereas H, SPY, and SPYH do not—demonstrating counterexamples for each. Our framework is both mathematically rigorous and extensible; empirical evaluation in protocol conformance testing confirms its effectiveness in detecting large-scale faults, significantly enhancing practical applicability in industrial settings.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

A fault domain that has been widely studied in black-box conformance testing is the class of finite state machines (FSMs) with at most $k$ extra states. Numerous methods for generating test suites have been proposed that guarantee fault coverage for this class. These test suites grow exponentially in $k$, so one can only run them for small $k$. But the assumption that $k$ is small is not realistic in practice. As a result, completeness for this fault domain has limited practical significance. As an alternative, we propose (much larger) fault domains that capture the assumption that when bugs in an implementation introduce extra states, these states can be reached via a few (at most $k$) transitions from states reachable via a set $A$ of common scenarios. Preliminary evidence suggests these fault domains, which contain FSMs with an exponential number of extra states (in $k$), are of practical use for testing network protocols. We present a sufficient condition for emph{$k$-$A$-completeness} of test suites with respect to these fault domains, phrased entirely in terms of properties of their testing tree. Our condition implies $k$-$A$-completeness of two prominent test suite generation algorithms, the Wp and HSI methods. Counterexamples show that three other approaches, the H, SPY and SPYH methods, do not always generate $k$-$A$-complete test suites.

Problem

• Addressing exponential growth in m-complete test suites
• Proposing larger fault domains with k-A-completeness condition
• Solving Hennie's challenge for finite state machine testing

Innovation

• Larger fault domains with k-A-completeness
• Sufficient condition for test suite completeness
• Wp and HSI methods solve Hennie's challenge