CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation

2026-04-14

AI Summary

It remains unclear whether current large language models genuinely understand program semantics in code generation, and there is a lack of systematic evaluation of their ability to generate executable behavioral specifications. This work proposes CodeSpecBench, the first benchmark supporting multi-granularity tasks at both function and repository levels, which expresses preconditions and postconditions as executable Python functions and emphasizes the correctness and completeness of specifications. Using an execution-driven protocol, it evaluates a model’s capacity to accept valid behaviors and reject invalid ones. Experiments across 15 state-of-the-art models reveal that the highest pass rate on repository-level tasks is only 20.2%, significantly lagging behind general code generation performance, thereby demonstrating that specification generation is substantially more challenging and that strong code generation capabilities do not equate to deep semantic understanding.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) can generate code from natural language, but the extent to which they capture intended program behavior remains unclear. Executable behavioral specifications, defined via preconditions and postconditions, provide a concrete means to assess such understanding. However, existing work on specification generation is constrained in evaluation methodology, task settings, and specification expressiveness. We introduce CodeSpecBench, a benchmark for executable behavioral specification generation under an execution-based evaluation protocol. CodeSpecBench supports both function-level and repository-level tasks and encodes specifications as executable Python functions. Constructed from diverse real-world codebases, it enables a realistic assessment of both correctness (accepting valid behaviors) and completeness (rejecting invalid behaviors). Evaluating 15 state-of-the-art LLMs on CodeSpecBench, we observe a sharp performance degradation on repository-level tasks, where the best model attains only a 20.2% pass rate. We further find that specification generation is substantially more challenging than code generation, indicating that strong coding performance does not necessarily reflect deep understanding of intended program semantics. Our data and code are available at https://github.com/SparksofAGI/CodeSpecBench.

Problem

• executable behavioral specification
• large language models
• program semantics
• specification generation
• code understanding

Innovation

• executable behavioral specifications
• LLM benchmarking
• preconditions and postconditions
• repository-level specification
• execution-based evaluation