IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation

2026-04-16

AI Summary

This work addresses the challenge of effectively quantifying uncertainty in large language models (LLMs) when generating long texts, particularly for semantically coherent yet factually incorrect statements. To this end, the authors propose the first claim-level uncertainty quantification framework based on a question-answering paradigm, employing an “interrogate-then-respond” mechanism that jointly evaluates factual accuracy through inter-sample consistency and intra-sample faithfulness. The approach is model-agnostic and thus broadly applicable across diverse LLMs without requiring architectural modifications. Experimental results demonstrate that the proposed method significantly outperforms existing techniques on two widely used long-form text generation benchmarks, exhibiting strong robustness and consistent effectiveness across different models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Despite the rapid advancement of Large Language Models (LLMs), uncertainty quantification in LLM generation is a persistent challenge. Although recent approaches have achieved strong performance by restricting LLMs to produce short or constrained answer sets, many real-world applications require long-form and free-form text generation. A key difficulty in this setting is that LLMs often produce responses that are semantically coherent yet factually inaccurate, while the underlying semantics are multifaceted and the linguistic structure is complex. To tackle this challenge, this paper introduces Interrogative Uncertainty Quantification (IUQ), a novel framework that leverages inter-sample consistency and intra-sample faithfulness to quantify the uncertainty in long-form LLM outputs. By utilizing an interrogate-then-respond paradigm, our method provides reliable measures of claim-level uncertainty and the model's faithfulness. Experimental results across diverse model families and model sizes demonstrate the superior performance of IUQ over two widely used long-form generation datasets. The code is available at https://github.com/louisfanhz/IUQ.

Problem

• uncertainty quantification
• large language models
• long-form generation
• factual accuracy
• free-form text

Innovation

• Interrogative Uncertainty Quantification
• long-form generation
• uncertainty quantification
• faithfulness
• inter-sample consistency