BeSpec: Behavior-Level Specification Alignment for Code Generation

2026-07-03

AI Summary

This work addresses the challenge that natural language specifications for programming tasks are often ambiguous or incomplete, leading large language models to generate code that is logically consistent yet deviates from intended behavior. To mitigate this, the authors propose BeSpec, a novel approach that introduces behavior-level specification alignment in code generation. BeSpec treats the task description as partial evidence of program behavior, constructs a verifiable behavioral model, and dynamically refines the specification or filters candidate programs by executing them on probe inputs and comparing observed outputs against expected behaviors. Integrating large language models, behavioral property modeling, execution feedback, and specification refinement, BeSpec establishes a behavior-consistency-driven generate-and-verify loop. Experiments demonstrate that BeSpec significantly outperforms nine baselines across four benchmarks, achieving Pass@1 improvements of 8.1%–25.3%; error analysis indicates that remaining failures primarily stem from algorithmic complexity rather than misinterpretation of specifications.

Abstract

LLMs have made substantial progress on automated code generation from natural-language descriptions of desired behavior (intent). Most existing methods improve generated programs through execution-guided code refinement: they generate a candidate solution, execute it, and patch the implementation using feedback, while leaving the underlying specification unchanged. This workflow implicitly assumes that the LLM's understanding of the intent is already correct and complete. In practice, however, intents are often ambiguous or underspecified. As a result, even a capable model may produce a correct implementation of the wrong intent, making specification mismatch a central bottleneck. This paper presents BeSpec, a behavioral model-based approach to specification alignment. BeSpec treats the task description as partial evidence about the intended behavior of the correct program. It first builds an explicit behavioral model, which are checkable properties that valid outputs must satisfy. BeSpec then generates candidate programs, executes them on probe inputs, and compares their observed behavior with the predicted behaviors. When observed behavior does not match the predicted behaviors, BeSpec either refines the specification or rejects the candidate program. We evaluate BeSpec with three LLMs on four benchmarks: CodeContests, xCodeEval, APPS, and the contamination-free LiveCodeBench. Against nine baselines, BeSpec achieves the highest Pass@1 and average pass rate across all settings, improving average Pass@1 over the strongest baseline by 8.1%--25.3% relative across the three LLMs. A failure analysis shows that after alignment, most remaining errors stem from algorithmic difficulty rather than misunderstood specifications, while ablation studies confirm that each major component of BeSpec contributes positively.

Problem

• specification alignment
• code generation
• intent ambiguity
• behavioral specification
• LLM

Innovation

• specification alignment
• behavioral model
• code generation
• LLM
• execution-guided refinement