Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs

2026-07-13

AI Summary

Existing black-box uncertainty methods struggle to capture semantic reliability in code generation and fail to distinguish trustworthy outputs from random guesses. This work proposes the first purely black-box framework that constructs semantic interaction graphs by executing generated code and introduces von Neumann entropy to quantify the global semantic diversity of the solution space, thereby assessing model uncertainty. By incorporating runtime behavior into black-box uncertainty estimation, the method bridges the gap between syntactic variation and semantic consistency. Experiments across eight prominent large code models demonstrate that the proposed uncertainty scores exhibit a strong negative correlation with functional correctness (Spearman ρ = −0.98), significantly outperforming lexical and embedding-based baselines and effectively enabling risk detection and selective prediction.

Abstract

As Code Large Language Models (LLMs) become central to modern software engineering, their inherent stochasticity poses significant real-world risks, where even minor errors can lead to severe functional, security, or safety consequences. Reliable automation, therefore, demands the ability to distinguish between confident, well-supported predictions and stochastic guessing. However, existing uncertainty estimation methods face a critical gap: white and grey-box techniques are often inapplicable to closed-source models, while standard "black-box" text metrics fail to capture the unique fragility of code, where syntactic variation does not always imply semantic divergence. To bridge this syntax-semantics gap, we introduce Code-MUE, a purely black-box framework that measures uncertainty through execution-based Semantic Interaction Graphs. Unlike prior approaches that rely on superficial textual similarity, Code-MUE grounds uncertainty in observable runtime behavior, calculating the Von Neumann entropy of the solution space to quantify global semantic diversity. A large-scale empirical study across eight state-of-the-art LLMs demonstrates that Code-MUE achieves a strong negative correlation with functional correctness (Spearman's correlation up to -0.98), significantly outperforming lexical and embedding-based baselines while enabling robust risk detection and selective prediction in practical workflows.

Problem

• Code LLMs
• uncertainty estimation
• black-box evaluation
• semantic divergence
• syntax-semantics gap

Innovation

• Code-MUE
• uncertainty estimation
• semantic interaction graphs
• execution-based evaluation
• Von Neumann entropy