How Low Can We Go? Minimizing Interaction Samples for Configurable Systems

2025-01-12

AI Summary

In testing configurable systems, existing t-wise interaction coverage approaches suffer from excessively large sample sizes, reliance on heuristics, low efficiency, and lack of theoretical guarantees. Method: This paper proposes a sampling optimization framework grounded in mathematical programming duality theory. It jointly computes provable lower bounds on sample size and near-optimal solutions for t-wise coverage, enabling solution quality certification and global optimality verification. Technically, it integrates dual modeling, a local search–enhanced algorithm (SampLNS), and constraint satisfaction optimization. Contribution/Results: Evaluated on 47 benchmark systems, the framework reduces sample sizes in 85% of cases and achieves—while formally proving—global optimality in 63%. It significantly lowers testing costs and manual tuning effort, establishing a new theoretical benchmark for minimal sample construction in configurable system testing.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Constraint Satisfaction and Optimization: Constraint Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Modern software systems are typically configurable, a fundamental prerequisite for wide applicability and reusability. This flexibility poses an extraordinary challenge for quality assurance, as the enormous number of possible configurations makes it impractical to test each of them separately. This is where t-wise interaction sampling can be used to systematically cover the configuration space and detect unknown feature interactions. Over the last two decades, numerous algorithms for computing small interaction samples have been studied, providing improvements for a range of heuristic results; nevertheless, it has remained unclear how much these results can still be improved. We present a significant breakthrough: a fundamental framework, based on the mathematical principle of duality, for combining near-optimal solutions with provable lower bounds on the required sample size. This implies that we no longer need to work on heuristics with marginal or no improvement, but can certify the solution quality by establishing a limit on the remaining gap; in many cases, we can even prove optimality of achieved solutions. This theoretical contribution also provides extensive practical improvements: Our algorithm SampLNS was tested on 47 small and medium-sized configurable systems from the existing literature. SampLNS can reliably find samples of smaller size than previous methods in 85% of the cases; moreover, we can achieve and prove optimality of solutions for 63% of all instances. This makes it possible to avoid cumbersome efforts of minimizing samples by researchers as well as practitioners, and substantially save testing resources for most configurable systems.

Problem

• Software Testing
• Interaction Examples Reduction
• Systematic Adjustment

Innovation

• SampLNS algorithm
• t-wise interaction sampling
• optimality proof