Synthesis Benchmarks for Automated Reasoning

2025-07-26

AI Summary

Existing automated program synthesis research lacks standardized benchmark datasets in the ∀∃ logical format that support uncomputable symbolic constraints. Method: We introduce the first ∀∃-logic benchmark suite tailored for deductive synthesis, formally modeling uncomputable symbolic restrictions, systematically extending classical instances, and incorporating a dynamic expansion mechanism. Our approach precisely encodes input-output specifications as ∀∃ formulas, integrating established benchmarks with newly constructed synthesis problems to ensure semantic rigor and task diversity. Contribution/Results: The resulting dataset features a clear hierarchical structure and strong extensibility, filling a critical gap in standardized evaluation infrastructure for deductive synthesis. It provides a unified, reliable foundation for rigorously assessing both the theoretical capabilities and practical effectiveness of deductive synthesis algorithms—particularly their ability to handle quantified, non-computable constraints—thereby advancing principled evaluation in program synthesis research.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Program synthesis is the task of constructing a program conforming to a given specification. We focus on deductive synthesis, and in particular on synthesis problems with specifications given as $forallexists$-formulas, expressing the existence of an output corresponding to any input. So far there has been no canonical benchmark set for deductive synthesis using the $forallexists$-format and supporting the so-called uncomputable symbol restriction. This work presents such a data set, composed by complementing existing benchmarks by new ones. Our data set is dynamically growing and should motivate future developments in the theory and practice of automating synthesis.

Problem

• Lack of canonical benchmarks for deductive program synthesis
• Need for ∀∃-formula specifications in synthesis problems
• Supporting uncomputable symbol restriction in synthesis benchmarks

Innovation

• Deductive synthesis with $forallexists$-formulas
• Uncomputable symbol restriction support
• Dynamic benchmark dataset for synthesis