POSTCONDBENCH: Benchmarking Correctness and Completeness in Formal Postcondition Inference

2026-05-05

AI Summary

This work addresses the lack of evaluation benchmarks that jointly assess correctness and completeness in automated postcondition generation for formal specification. The authors construct a multilingual benchmark derived from real-world software projects, introducing—for the first time—a completeness metric based on defect detection, accompanied by an expert-annotated postcondition dataset. The benchmark integrates program execution validation, multilingual static analysis, and a large language model (LLM) evaluation framework to enable fully automated assessment. Experiments across 420 Python and Java tasks reveal a significant gap between state-of-the-art LLMs’ performance in correctness versus completeness, with results further influenced substantially by project dependencies and method complexity.

Abstract

Formal postconditions precisely characterize program behavior and support debugging, testing, and verification, but writing them requires substantial expertise and effort. This has motivated recent work on automatically generating postconditions from code and natural-language artifacts using large language models (LLMs). However, evaluation remains a key bottleneck. Existing benchmarks primarily emphasize correctness under limited evaluation settings, often relying on surface-form matching or manual assessment on small or synthetic datasets. We introduce POSTCONDBENCH, a multilingual benchmark for evaluating method-level postcondition generation from real-world software. POSTCONDBENCH comprises 420 Python and Java tasks drawn from 121 open-source projects, each paired with a high-quality ground-truth postcondition set constructed with expert involvement. To enable automatic evaluation, POSTCONDBENCH provides a runnable execution environment and operationalizes completeness via defect discrimination: a postcondition set is more complete if it is violated by more defective implementations while remaining satisfied on correct executions. Using POSTCONDBENCH, we formulate three generation settings and evaluate five SOTA LLMs. Our results reveal a substantial gap between correctness and completeness, and show that repository-level dependencies and method complexity exacerbate this gap.

Problem

• postcondition inference
• formal verification
• benchmarking
• completeness
• correctness

Innovation

• postcondition inference
• formal specification
• LLM evaluation
• defect discrimination
• software correctness