Improved Confidence Estimates for Black-Box Large Language Models

2026-08-19

AI Summary

本文通过构建分类器预测大语言模型响应正确性，利用现有评分和相似查询的正确性作为特征，提高不确定性量化表现。

Abstract

Uncertainty quantification (UQ) is essential for the safe deployment of large language models (LLMs). Existing methods, from verbalized confidence to ones requiring multiple generations, are often zero-shot and produce scores quantifying uncertainty without the need for labelled data. Nonetheless, in practice one must always evaluate their performance on a dataset of interest before deployment. In this work we show that, by leveraging this dataset, we consistently outperform these existing scores. Specifically, we build simple classifiers that predict LLM response correctness by using these scores and the correctness of similar queries as features. Our method produces minimal computational overhead, making it a cheap and straightforward enhancement for UQ in LLMs for real-world applications.

Problem

• Uncertainty Quantification
• Large Language Models
• Confidence Estimates

Innovation

• Uncertainty Quantification
• Large Language Models
• Classifier
• Computational Overhead