Requirements Coverage-Guided Minimization for Natural Language Test Cases

2025-05-26

AI Summary

Requirement-based test suites derived from natural language specifications often contain redundancy, increasing testing costs and degrading fault detection rate (FDR). Method: We propose RTM, a coverage-guided test suite minimization approach. RTM introduces the first requirement-coverage-driven genetic algorithm framework, integrating coverage-preserving initialization with multi-granularity textual representations—including three preprocessing schemes, seven embedding models, and three distance metrics—to guarantee 100% requirement coverage under strict budget constraints while maximizing FDR. Contribution/Results: Evaluated on an automotive industry dataset, RTM significantly improves FDR without sacrificing full requirement coverage. Further analysis reveals that redundancy level is a critical determinant of test suite minimization (TSM) effectiveness. This work establishes a novel, interpretable, and reproducible paradigm for efficient TSM in requirement validation scenarios.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

As software systems evolve, test suites tend to grow in size and often contain redundant test cases. Such redundancy increases testing effort, time, and cost. Test suite minimization (TSM) aims to eliminate such redundancy while preserving key properties such as requirement coverage and fault detection capability. In this paper, we propose RTM (Requirement coverage-guided Test suite Minimization), a novel TSM approach designed for requirement-based testing (validation), which can effectively reduce test suite redundancy while ensuring full requirement coverage and a high fault detection rate (FDR) under a fixed minimization budget. Based on common practice in critical systems where functional safety is important, we assume test cases are specified in natural language and traced to requirements before being implemented. RTM preprocesses test cases using three different preprocessing methods, and then converts them into vector representations using seven text embedding techniques. Similarity values between vectors are computed utilizing three distance functions. A Genetic Algorithm, whose population is initialized by coverage-preserving initialization strategies, is then employed to identify an optimized subset containing diverse test cases matching the set budget. We evaluate RTM on an industrial automotive system dataset comprising $736$ system test cases and $54$ requirements. Experimental results show that RTM consistently outperforms baseline techniques in terms of FDR across different minimization budgets while maintaining full requirement coverage. Furthermore, we investigate the impact of test suite redundancy levels on the effectiveness of TSM, providing new insights into optimizing requirement-based test suites under practical constraints.

Problem

• Reduces test suite redundancy while maintaining requirement coverage
• Optimizes test cases using NLP and genetic algorithms
• Improves fault detection rate in industrial automotive systems

Innovation

• Uses Genetic Algorithm for test suite optimization
• Employs multiple text embedding techniques
• Ensures full requirement coverage and high FDR