GENUINE: Graph Enhanced Multi-level Uncertainty Estimation for Large Language Models

2025-09-09

AI Summary

Existing large language models predominantly estimate uncertainty via token-level probabilities, neglecting semantic and syntactic structural dependencies—leading to unreliable confidence estimates in high-stakes scenarios. To address this, we propose the first syntax-aware, multi-level uncertainty modeling framework: it introduces dependency parse trees into uncertainty quantification for the first time, integrating graph neural networks with hierarchical graph pooling to explicitly model structure–semantics correlations under a supervised learning paradigm. Evaluated across multiple NLP tasks, our method significantly improves uncertainty calibration—achieving up to a 29% increase in AUROC and over a 15% reduction in Expected Calibration Error (ECE)—outperforming semantic-entropy-based and other baseline approaches. This work establishes a novel, structure-aware paradigm for trustworthy text generation.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Uncertainty estimation is essential for enhancing the reliability of Large Language Models (LLMs), particularly in high-stakes applications. Existing methods often overlook semantic dependencies, relying on token-level probability measures that fail to capture structural relationships within the generated text. We propose GENUINE: Graph ENhanced mUlti-level uncertaINty Estimation for Large Language Models, a structure-aware framework that leverages dependency parse trees and hierarchical graph pooling to refine uncertainty quantification. By incorporating supervised learning, GENUINE effectively models semantic and structural relationships, improving confidence assessments. Extensive experiments across NLP tasks show that GENUINE achieves up to 29% higher AUROC than semantic entropy-based approaches and reduces calibration errors by over 15%, demonstrating the effectiveness of graph-based uncertainty modeling. The code is available at https://github.com/ODYSSEYWT/GUQ.

Problem

• Improving LLM reliability via uncertainty estimation
• Capturing semantic dependencies in generated text
• Enhancing structural relationship modeling for confidence

Innovation

• Leverages dependency parse trees for structure
• Uses hierarchical graph pooling techniques
• Incorporates supervised learning for relationships