The Origins of Stochasticity: Comprehensive Investigations on Uncertainty Quantification for Large Language Models

2026-06-21

AI Summary

This work addresses the challenge of quantifying uncertainty in large language models (LLMs), which arises from multiple intertwined sources and is inadequately captured by existing methods. The authors propose a fine-grained uncertainty taxonomy that decomposes uncertainty into four dimensions—input, parameters, tokens, and decoding—and introduce a provenance-aware framework spanning the entire generation process. Through systematic evaluation across prominent models including Qwen3, Llama 3.2, and DeepSeek-V3, they find that consensus-based approaches (e.g., Deg, EigV) consistently outperform alternatives such as Bayesian inference, ensembles, and single-pass methods. Moreover, increasing model scale substantially reduces uncertainty, following an empirical scaling law, while the effectiveness of uncertainty quantification techniques exhibits strong task dependence, with marked performance variations observed on benchmarks like TriviaQA, GSM8K, and HumanEval.

Abstract

Recent advancements in Large Language Models (LLMs) have enabled sophisticated reasoning and content generation, yet their inherent stochasticity poses significant challenges for ensuring predictive credibility. While traditional uncertainty taxonomy paradigms, such as the dichotomy of aleatoric and epistemic uncertainties, provide conceptual foundations, they often fail to capture the multi-component and multi-stage nature of LLM generation and struggle to evaluate the effectiveness of various Uncertainty Quantification (UQ) methods. In this paper, we propose a granular uncertainty taxonomy that systematically attributes LLM uncertainty into input-level, parameter-level, token-level, and decoding-process sources. Correspondingly, we categorize existing UQ methods into Bayesian, ensemble, consensus-based, and single-pass approaches. Furthermore, we introduce a comprehensive evaluation framework covering diverse generation settings and metrics. We empirically evaluate 21 typical UQ methods across three prominent LLM families, including Qwen3, Llama 3.2, and DeepSeek-V3, on benchmarks such as TriviaQA, GSM8K, and HumanEval. Our experimental results demonstrate that (i) the effectiveness of UQ methods is sensitive to task types and generation settings; (ii) consensus-based methods, typed Deg and EigV, consistently outperform other UQ approaches; and (iii) larger model scales correlate with lower uncertainty estimates, suggesting an empirical scaling law for LLM uncertainty. This work bridges the gap between theoretical origins and practical deployment, providing a versatile diagnostic tool for systematically quantifying uncertainty in LLM applications.

Problem

• stochasticity
• uncertainty quantification
• large language models
• uncertainty taxonomy
• predictive credibility

Innovation

• uncertainty quantification
• large language models
• stochasticity
• consensus-based methods
• evaluation framework