Do LLMs Trust the Code They Write?

2025-12-08

AI Summary

Large language models (LLMs) exhibit a decoupling between output token probabilities and actual code correctness, limiting reliability in AI-assisted programming. Method: We investigate whether internal hidden states encode implicit representations of code correctness, proposing a contrastive hidden-state analysis framework to probe such signals across multiple state-of-the-art LLMs. Contribution/Results: We empirically discover—and for the first time extract—a robust, execution-free correctness signal from layer-wise hidden states that reliably discriminates correct from incorrect code. This signal requires neither program execution nor human annotation, and substantially outperforms conventional log-probability scores and verbalized confidence estimates. In code quality ranking and sample filtering tasks, it achieves an average accuracy improvement of 12.7%. Our findings establish a new paradigm for building high-fidelity, execution-free AI programming systems and provide a foundational technical pathway toward trustworthy code generation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Reasoning under Uncertainty: Probabilistic Programming

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Despite the effectiveness of large language models (LLMs) for code generation, they often output incorrect code. One reason is that model output probabilities are often not well-correlated with correctness, and reflect only the final output of the generation process. Inspired by findings that LLMs internally encode concepts like truthfulness, this paper explores if LLMs similarly represent code correctness. Specifically, we identify a correctness representation inside LLMs by contrasting the hidden states between pairs of correct and incorrect code for the same programming tasks. By experimenting on four LLMs, we show that exploiting this extracted correctness representation outperforms standard log-likelihood ranking, as well as verbalized model confidence. Furthermore, we explore how this internal correctness signal can be used to select higher-quality code samples, without requiring test execution. Ultimately, this work demonstrates how leveraging internal representations can enhance code generation systems and make LLMs more reliable, thus improving confidence in automatically generated code.

Problem

• LLMs often generate incorrect code despite high probabilities
• Internal correctness representation can improve code quality selection
• Leveraging hidden states enhances reliability without test execution

Innovation

• Extract correctness representation from hidden states
• Contrast correct and incorrect code pairs
• Select high-quality code without test execution