Programmable Property-Based Testing

2026-02-20

AI Summary

This work addresses the tight coupling between property definition and execution in existing property-based testing frameworks, which severely limits user customizability. To overcome this limitation, the authors propose a hygienic embedded language based on delayed-binding abstract syntax that represents properties as manipulable data structures, thereby decoupling property specification from execution logic. The design is implemented in both a dependently typed language (Rocq) and a dynamically typed language (Racket), enabling reifiable property representations and flexible execution strategies. Experimental results demonstrate that this approach facilitates rapid prototyping of diverse property runners, significantly enhancing the programmability, flexibility, and domain adaptability of the testing process.

Technology Category

• Reasoning under Uncertainty: Probabilistic Programming
• Knowledge Representation and Reasoning: Knowledge Representation Languages
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Property-based testing (PBT) is a popular technique for establishing confidence in software, where users write properties -- i.e., executable specifications -- that can be checked many times in a loop by a testing framework. In modern PBT frameworks, properties are usually written in shallowly embedded domain-specific languages, and their definition is tightly coupled to the way they are tested. Such frameworks often provide convenient configuration options to customize aspects of the testing process, but users are limited to precisely what library authors had the prescience to allow for when developing the framework; if they want more flexibility, they may need to write a new framework from scratch. We propose a new, deeper language for properties based on a mixed embedding that we call deferred binding abstract syntax, which reifies properties as a data structure and decouples them from the property runners that execute them. We implement this language in Rocq and Racket, leveraging the power of dependent and dynamic types, respectively. Finally, we showcase the flexibility of this new approach by rapidly prototyping a variety of property runners, highlighting domain-specific testing improvements that can be unlocked by more programmable testing.

Problem

• property-based testing
• programmable testing
• framework flexibility
• shallow embedding
• testing customization

Innovation

• property-based testing
• deferred binding abstract syntax
• programmable testing
• mixed embedding
• test framework extensibility