On the Role of Unobserved Sequences on Sample-based Uncertainty Quantification for LLMs

2025-10-05

AI Summary

This work addresses a long-overlooked challenge in uncertainty quantification for large language models (LLMs): the substantial bias introduced by unobserved output sequences—i.e., valid continuations not sampled during estimation. Existing sampling-based methods (e.g., entropy estimation) routinely neglect the probability mass assigned to such sequences, severely limiting hallucination detection. To remedy this, we formally model the latent output distribution and explicitly incorporate the probability mass of unobserved sequences into uncertainty estimation. We propose a principled framework for improved uncertainty quantification that accounts for this missing mass. Experiments reveal that ignoring unobserved sequences underestimates entropy by an average of 38%, degrading high-risk output identification. Our method improves hallucination detection F1-score by up to 12.6% across multiple benchmarks, offering both theoretical grounding and practical tools for deploying trustworthy LLMs in safety-critical applications.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Quantifying uncertainty in large language models (LLMs) is important for safety-critical applications because it helps spot incorrect answers, known as hallucinations. One major trend of uncertainty quantification methods is based on estimating the entropy of the distribution of the LLM's potential output sequences. This estimation is based on a set of output sequences and associated probabilities obtained by querying the LLM several times. In this paper, we advocate and experimentally show that the probability of unobserved sequences plays a crucial role, and we recommend future research to integrate it to enhance such LLM uncertainty quantification methods.

Problem

• Quantifying uncertainty in LLMs to detect hallucinations
• Estimating entropy of potential output sequences distribution
• Incorporating unobserved sequences probability for uncertainty quantification

Innovation

• Incorporates unobserved sequences probability in uncertainty quantification
• Enhances entropy estimation for LLM output distributions
• Improves sample-based uncertainty methods through unobserved sequences