Trace-Guided Synthesis of Effectful Test Generators

2026-04-05

AI Summary

This work addresses the challenge of efficiently testing black-box systems with side effects by proposing a test generation approach that integrates under-approximate typing with effect systems. The method employs symbolic traces to capture data and control dependencies of side-effecting operations, preserving essential constraints to guide test case synthesis. Precise coverage is achieved through an integration of property-based testing and model checking. The implemented tool, Clouseau, demonstrates substantial improvements over default strategies in frameworks such as QCheck and P, achieving test effectiveness comparable to state-of-the-art hand-crafted test suites. These results validate both the efficacy and practicality of the proposed methodology.

Abstract

Several recently proposed program logics have incorporated notions of underapproximation into their design, enabling them to reason about reachability rather than safety. In this paper, we explore how similar ideas can be integrated into an expressive type and effect system. We use the resulting underapproximate type specifications to guide the synthesis of test generators that probe the behavior of effectful black-box systems. A key novelty of our type language is its ability to capture underapproximate behaviors of effectful operations using symbolic traces that expose latent data and control dependencies, constraints that must be preserved by the test sequences the generator outputs. We implement this approach in a tool called Clouseau, and evaluate it on a diverse range of applications by integrating Clouseau's synthesized generators into property-based testing frameworks like QCheck and model-checking tools like P. In both settings, the generators synthesized by Clouseau are significantly more effective than the default testing strategy, and are competitive with state-of-the-art, handwritten solutions.

Problem

• test generation
• effectful systems
• black-box testing
• symbolic traces
• underapproximation

Innovation

• underapproximate type system
• symbolic traces
• effectful test generation
• program synthesis
• property-based testing