Towards Comprehensive Sampling of SMT Solutions

2025-11-13

AI Summary

Diverse solution generation for SMT formulas—over bit-vectors, arrays, and uninterpreted functions—is critical for fault detection and security verification in hardware/software testing, yet existing approaches struggle to balance constraint-space coverage with sampling efficiency. This paper introduces PanSampler, the first diversity-aware SMT sampling framework, integrating an AST-guided solution scoring mechanism, iterative sampling, and localized search-based optimization. Its key innovation lies in incorporating syntactic structural information into diversity assessment and applying targeted post-sampling refinement, thereby achieving a synergistic trade-off between high coverage and low sample count. Experimental evaluation demonstrates that PanSampler reduces the number of solutions required to achieve equivalent coverage by 32.6%–76.4% compared to state-of-the-art methods, significantly improving defect detection rates and testing efficiency.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

This work focuses on effectively generating diverse solutions for satisfiability modulo theories (SMT) formulas, targeting the theories of bit-vectors, arrays, and uninterpreted functions, which is a critical task in software and hardware testing. Generating diverse SMT solutions helps uncover faults and detect safety violations during the verification and testing process, resulting in the SMT sampling problem, i.e., constructing a small number of solutions while achieving comprehensive coverage of the constraint space. While high coverage is crucial for exploring system behaviors, reducing the number of solutions is of great importance, as excessive solutions increase testing time and resource usage, undermining efficiency. In this work, we introduce PanSampler, a novel SMT sampler that achieves high coverage with a small number of solutions. It incorporates three novel techniques, i.e., diversity-aware SMT algorithm, abstract syntax tree (AST)-guided scoring function and post-sampling optimization technology, enhancing its practical performance. It iteratively samples solutions, evaluates candidates, and employs local search to refine solutions, ensuring high coverage with a small number of samples. Extensive experiments on practical benchmarks demonstrate that PanSampler exhibits a significantly stronger capability to reach high target coverage, while requiring fewer solutions than current samplers to achieve the same coverage level. Furthermore, our empirical evaluation on practical subjects, which are collected from real-world software systems, shows that PanSampler achieves higher fault detection capability and reduces the number of required test cases from 32.6% to 76.4% to reach the same fault detection effectiveness, leading to a substantial improvement in testing efficiency. PanSampler advances SMT sampling, reducing the cost of software testing and hardware verification.

Problem

• Generating diverse SMT solutions for software/hardware verification
• Achieving comprehensive constraint space coverage with minimal samples
• Reducing testing resource usage while maintaining fault detection capability

Innovation

• Diversity-aware SMT algorithm for solution generation
• AST-guided scoring function to evaluate candidates
• Post-sampling optimization technology refining solutions