Evaluating LLM Uncertainty in Long-Form Generation Using Deterministic Ground Truth

2026-07-04

AI Summary

This work addresses the limitations of existing uncertainty estimation methods for long-form text generation, which struggle to pinpoint token-level errors and rely on noisy human annotations. The authors propose SALT—the first fine-grained evaluation benchmark grounded in deterministic ground truth—spanning six procedurally generated tasks and enabling assessment of correctness, calibration, and confidence ranking from atomic units to full documents. Through systematic analysis of over 50 large language models, the study reveals that confidence ranking consistently fails at the atomic level and identifies two key error sources: prefix contamination propagation and performance degradation induced by extended context lengths. SALT enables fine-grained error detection and risk control without external evaluators, establishing a new paradigm for reliability assessment in long-text generation.

Abstract

As LLMs generate increasingly long outputs, effective uncertainty estimation must identify errors at fine-grained levels rather than discard entire responses. While such methods exist, evaluating uncertainty at any resolution (token to an entire generation) is challenging and highly sensitive to label imperfections, making zero-noise benchmarks essential; yet, long-form generation benchmarks tend to rely on fallible labels rather than deterministic ground truth. We introduce Single-answer Atomic Long-form Target (SALT), a benchmark of six procedurally generated tasks with single deterministic long textual ground truths, enabling unit-level evaluation of correctness, calibration, and ranking without external judges. Equipped with SALT, our analysis of 50+ LLMs reveals key insights: We identify which confidence functions dominate each uncertainty aspect and show that confidence ranking largely breaks at atomic resolution, even when clearer separability emerges at coarser line-level units. SALT further enables controlled atom-level interventions throughout generation, revealing two separable drivers of future errors: propagation from corrupted prefixes, dominated by global context correctness, and bounded degradation from increasing answer-context length. Finally, we demonstrate that reasoning, via Chain-of-Thought prompting or internalized through training, introduces a trade-off, improving accuracy while degrading confidence ranking. These findings directly impact risk-critical applications requiring reliable error identification and mitigation.

Problem

• LLM uncertainty
• long-form generation
• deterministic ground truth
• fine-grained evaluation
• confidence calibration

Innovation

• uncertainty estimation
• long-form generation
• deterministic ground truth
• fine-grained evaluation
• confidence calibration