Eigenvalue Calibration for Semantic Embeddings of Large Language Models

2026-07-09

AI Summary

This work addresses the lack of effective calibration methods for quantifying uncertainty in semantic embeddings produced by large language models, where conventional classification-based probability calibration techniques are inapplicable. The paper introduces the first calibration framework tailored to the eigenvalues of semantic embedding representations, modeling the model’s output as a density matrix and applying temperature scaling directly to its eigenvalues. Theoretical analysis establishes an equivalence between post-calibration entropy and proper scoring risk, derives an eigenvalue-specific centered calibration inequality, and proves that the proposed method achieves optimal calibration under minimal proper scoring risk. Experiments reveal pervasive overconfidence in current models and demonstrate that the proposed approach substantially improves calibration performance of semantic embeddings, confirming both its theoretical soundness and practical utility.

Abstract

Uncertainty quantification is central to the reliable deployment of large language models (LLMs), and eigenvalues of semantic embeddings have recently emerged as a key tool in state-of-the-art methods. However, conventional calibration results developed for classification probabilities cannot be directly transferred to eigenvalues. We address this gap by proposing a novel framework for calibrating the eigenvalues of semantic embeddings. We interpret LLMs combined with semantic embeddings of their generated answers as density matrix predictors, and we propose a novel approach to calibrate density matrix predictors by applying temperature scaling to their eigenvalues. We establish entropy-risk equivalence under calibration, derive a central calibration inequality specific to eigenvalues, and prove that temperature-scaled eigenvalues optimize calibration when minimizing proper score risks. Experiments on a variety of real-world settings show that current LLMs are systematically overconfident, and validate our theoretical findings. Together, these results advance the foundations and practice of uncertainty quantification for semantic embeddings.

Problem

• eigenvalue calibration
• semantic embeddings
• large language models
• uncertainty quantification
• density matrix predictors

Innovation

• eigenvalue calibration
• semantic embeddings
• temperature scaling
• density matrix predictors
• uncertainty quantification