We've Got You Covered: Type-Guided Repair of Incomplete Input Generators

2025-04-08

AI Summary

This paper addresses the problem that manually written input generators for property-based testing often fail to produce all valid inputs satisfying preconditions due to incomplete coverage, thereby reducing test effectiveness. To resolve this, the authors propose a type-guided automatic repair method for such generators. Their core contribution is the first enumeration-based program synthesis algorithm grounded in *coverage types*—a novel type system that formally characterizes the semantic coverage relationship between missing input subspaces and candidate repairs, ensuring semantic completeness of the repaired generator. Implemented in OCaml as the prototype tool Cobb, the approach successfully repairs diverse incomplete generators on standard benchmarks. Evaluation shows substantial improvements in both input coverage and property verification pass rates, demonstrating the method’s efficacy and practicality.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

Property-based testing is a popular technique for automatically testing semantic properties of a program, specified as a pair of pre- and post-conditions. The efficacy of this approach depends on being able to quickly generate inputs that meet the precondition, in order to maximize the set of program behaviors that are probed. For semantically rich preconditions, purely random generation is unlikely to produce many valid inputs; when this occurs, users are forced to manually write their own specialized input generators. One common problem with handwritten generators is that they may be emph{incomplete}, i.e., they are unable to generate some values meeting the target precondition. This paper presents a novel program repair technique that patches an incomplete generator so that its range includes every valid input. Our approach uses a novel enumerative synthesis algorithm that leverages the recently developed notion of emph{coverage types} to characterize the set of missing test values as well as the coverage provided by candidate repairs. We have implemented a repair tool for OCaml input generators, called Cobb, and have used it to repair a suite of benchmarks drawn from the property-based testing literature.

Problem

• Repairing incomplete handwritten input generators
• Ensuring generators cover all valid inputs
• Using coverage types for enumerative synthesis

Innovation

• Type-guided repair for incomplete generators
• Enumerative synthesis with coverage types
• Automated repair tool for OCaml generators