When is LLM-Based Program Reasoning Correct? A Completion Semantics for LLM-Based Code Inference

2026-07-14

AI Summary

This work addresses the tendency of large language models (LLMs) to rely on implicit assumptions about missing context in program reasoning, which often renders their conclusions unverifiable. It introduces, for the first time, a formal completion semantics framework tailored to LLMs, treating incomplete programs as spaces of possible completions. The approach validates whether an LLM’s reasoning rests on plausible assumptions by generating executable “witness” instances that concretize these assumptions. Integrating formal semantic modeling, completion space construction, and witness generation, the method effectively distinguishes sound from unrealistic reasoning in real-world tasks such as LLM-generated vulnerability reports and program analyses, thereby substantially enhancing the trustworthiness of program reasoning by LLMs.

Abstract

Due to token and cognitive limits, Large Language Models (LLMs) typically perform program reasoning over incomplete code fragments/prompts rather than complete programs. Such reasoning therefore must rely on {assumptions about omitted code and context. As a result, the meaning of an inference over a program fragment is not absolute, but depends on an implicit completion model describing how the fragment may be refined into a complete program. In this paper, we introduce completion semantics for LLM-based program reasoning. We formalize incomplete programs as denoting a space of possible refinements and define the correctness of existential inferences relative to a completion model. Under this view, a reported bug is correct whenever there exists a completion within the model that witnesses the bug. This perspective explains why many LLM-generated reports are neither simply correct nor incorrect, but instead depend on assumptions about omitted context. We have instantiated our approach in the form of a witness-generation workflow that concretizes completions underlying an inference by constructing executable refinements of the original program fragment. Witnesses serve both as evidence for existential claims and as a mechanism for exposing the assumptions required to support them. We evaluate our approach on real-world LLM-generated bug reports and program-analysis tasks. Our results show that witness generation effectively distinguishes inferences supported by plausible completions from those requiring unrealistic assumptions, providing a practical mechanism for validating reasoning over incomplete programs.

Problem

• LLM-based program reasoning
• incomplete programs
• completion semantics
• correctness
• program fragments

Innovation

• completion semantics
• LLM-based program reasoning
• witness generation
• incomplete programs
• program refinement