Multicalibration for LLM-based Code Generation

2025-12-09

AI Summary

This work addresses the misalignment between confidence scores and actual correctness in code generation by large language models (LLMs). We propose the first multidimensional calibration framework tailored for code generation, introducing multicalibration—previously unexplored in the code domain—to enable conditional calibration across fine-grained attributes such as programming language, problem complexity, and generated code length. We implement four multicalibration methods on state-of-the-art models—including Qwen3 Coder, GPT-OSS, and DeepSeek-R1-Distill—within a function synthesis benchmark. Experimental results show that our approach improves skill score by 1.03 over the uncalibrated baseline and outperforms conventional calibration methods by 0.37. To foster reproducibility and further research, we publicly release the first standardized code calibration dataset, comprising generated code samples, likelihood estimates, and ground-truth correctness labels.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As AI-based code generation becomes widespread, researchers are investigating the calibration of code LLMs - ensuring their confidence scores faithfully represent the true likelihood of code correctness. To do so, we investigate multicalibration, which can capture additional factors about a coding problem, such as complexity, code length, or programming language used. We study four multicalibration approaches on three function synthesis benchmarks, using latest-generation code LLMs (Qwen3 Coder, GPT-OSS, DeepSeek-R1-Distill). Our results demonstrate that multicalibration can yield distinct improvements over both uncalibrated token likelihoods (+1.03 in skill score) and baseline calibrations (+0.37 in skill score). We study the influence of the aforementioned factors in ablations, and make our dataset (consisting of code generations, likelihoods, and correctness labels) available for future research on code LLM calibration.

Problem

• Ensuring code LLMs' confidence scores reflect true correctness likelihood
• Investigating multicalibration to capture coding problem factors like complexity
• Improving calibration over uncalibrated token likelihoods and baseline methods

Innovation

• Multicalibration improves code LLM confidence scores
• It incorporates problem complexity and code length
• Approach tested on latest code generation models