From Entropy to Calibrated Uncertainty: Training Language Models to Reason About Uncertainty

2026-03-06

AI Summary

This work addresses the challenge of enabling large language models to express uncertainty efficiently, interpretably, and reliably in high-stakes scenarios. To this end, the authors propose a novel three-stage post-training framework that uniquely integrates fine-grained entropy-based uncertainty scoring with Platt scaling and reinforcement learning, augmented by explicit modeling of the embedding space distribution. The resulting model produces well-calibrated and interpretable uncertainty estimates at inference time without requiring additional computation. Extensive evaluation demonstrates that the approach significantly outperforms existing baselines and exhibits strong generalization to unseen tasks, highlighting its robust uncertainty reasoning capabilities.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Multiagent Systems: Multiagent Systems under Uncertainty

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) that can express interpretable and calibrated uncertainty are crucial in high-stakes domains. While methods to compute uncertainty post-hoc exist, they are often sampling-based and therefore computationally expensive or lack calibration. We propose a three-stage pipeline to post-train LLMs to efficiently infer calibrated uncertainty estimates for their responses. First, we compute fine-grained entropy-based uncertainty scores on the training data, capturing the distributional variability of model outputs in embedding space. Second, these scores are calibrated via Platt scaling, producing reliable and human-interpretable uncertainty signals. Finally, the target LLM is post-trained via reinforcement learning to align its policy with these calibrated signals through a verifiable reward function. Unlike post-hoc uncertainty estimation methods, our approach provides interpretable and computationally efficient uncertainty estimates at test time. Experiments show that models trained with our pipeline achieve better calibration than baselines and generalize to unseen tasks without further processing, suggesting that they learn a robust uncertainty reasoning behavior.

Problem

• uncertainty calibration
• large language models
• interpretable uncertainty
• post-hoc uncertainty
• model reliability

Innovation

• calibrated uncertainty
• entropy-based scoring
• Platt scaling
• reinforcement learning
• uncertainty reasoning