Using Semantic Distance to Estimate Uncertainty in LLM-Based Code Generation

2026-05-09

AI Summary

Existing sampling-based uncertainty estimation methods struggle to effectively capture behavioral discrepancies in code generation by large language models and lack fine-grained characterization of execution semantics. This work proposes the first uncertainty estimation framework that integrates semantic distance, leveraging multi-candidate program sampling, execution trace analysis, and semantic distance metrics to accurately assess the reliability of generated code—without requiring access to internal model representations or invoking LLM-as-a-judge. The approach significantly outperforms existing techniques across multiple programming languages and benchmarks, reducing runtime by 48%–79% while maintaining robustness across diverse models and settings, thereby addressing a critical gap in quantifying behavioral divergence in code generation.

Abstract

LLMs show strong performance in code generation, but their outputs lack correctness guarantees. Sample-based uncertainty estimators address this by generating multiple candidate programs and measuring their disagreement. However, existing estimators make different design choices about how behaviours are identified, aggregated, referenced and compared, making them difficult to assess. We therefore first introduce a taxonomy that disentangles these choices and reveals a missing design point: semantic distance-aware uncertainty estimation, which measures not only whether sampled programs disagree, but how severely their execution behaviours differ. Across LiveCodeBench, MBPP, HumanEval-X and BigCodeBench, spanning Python, Java and C++, our metrics provide strong proxies for correctness, and consistently outperform state-of-the-art sample-based baselines across both closed-source models (GPT-3.5-Turbo, GPT-4o-mini, Gemini-2.5-Flash-Lite, Claude Opus 4.5) and an open-source model (DeepSeek-Coder-V2). The method is practical: it requires neither model internals nor LLM-as-judge calls, remains robust across models, languages, sampling temperatures and fuzzing settings, and reduces runtime by approximately 48-79% relative to existing baselines.

Problem

• uncertainty estimation
• code generation
• semantic distance
• large language models
• program correctness

Innovation

• semantic distance
• uncertainty estimation
• code generation
• LLM reliability
• sample-based evaluation