Synthesizing Test Cases for Narrowing Specification Candidates

2025-11-24

AI Summary

This work addresses the automated selection of formal specifications from a candidate set. The proposed test-driven specification discrimination method generates minimal discriminating test suites and dynamically prunes the candidate specification set based on user feedback. Methodologically, we design two solver-driven algorithms: one optimizes for minimal test suite size, while the other prioritizes scalability—efficiently handling dozens of Alloy specifications. Both integrate symbolic execution and constraint solving to systematically explore behavioral differences among candidates. Experimental evaluation demonstrates that the approach is both feasible and scalable across diverse real-world scenarios. It reliably identifies the unique specification consistent with intended behavior, thereby significantly enhancing the automation level and trustworthiness of specification selection in formal modeling.

Technology Category

• Constraint Satisfaction and Optimization: Solvers and Tools
• Search and Optimization: Sampling/Simulation-based Search
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

This paper proposes a technique to help choose the best formal specification candidate among a set of alternatives. Given a set of specifications, our technique generates a suite of test cases that, once classified by the user as desirable or not, narrows down the set of candidates to at most one specification. Two alternative solver-based algorithms are proposed, one that generates a minimal test suite, and another that does not ensure minimality. Both algorithms were implemented in a prototype that can be used generate test suites to help choose among alternative Alloy specifications. Our evaluation of this prototype against a large set of problems showed that the optimal algorithm is efficient enough for many practical problems, and that the non-optimal algorithm can scale up to dozens of candidate specifications while still generating reasonably sized test suites.

Problem

• Selecting optimal formal specification from multiple candidates
• Generating test cases to classify specification desirability
• Developing solver-based algorithms for specification narrowing

Innovation

• Generates test cases to select best specification
• Uses solver-based algorithms for test suite creation
• Implements both minimal and scalable non-minimal approaches