BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs

2026-08-31

AI Summary

为了解决大型语言模型在关键安全场景下的可靠性评估问题，提出了一种基于语义一致性的BiG-SURE方法，通过构建二分图来估计模型的不确定性。

Abstract

Reliable uncertainty estimation is a crucial requirement for deploying large language models (LLMs) and vision-language models (VLMs) in safety-critical settings, especially when the model parameters are not accessible (black-box). We propose BiG-SURE, an uncertainty estimator based on cross-temperature semantic agreement. The method samples low-temperature responses as stable semantic anchors and high-temperature responses as probes under meaning-preserving input transformations. It then constructs an anchor-probe Bipartite Graph (BiG) using NLI-based entailment scores and defines confidence through the normalized squared spectral energy of this matrix, with uncertainty given by its complement. This bipartite graph-based Semantic Uncertainty and Reliability Estimation (SURE) score measures whether high-temperature probes remain semantically aligned with the model's stable low-temperature belief or not. We evaluate BiG-SURE on text QA, multilingual QA, and multimodal QA tasks across multiple model families. In these experiments, BiG-SURE improves average abstention AUROC over prior black-box uncertainty estimators, while remaining simple, unsupervised, and applicable to black-box model settings.

Problem

• uncertainty estimation
• large language models
• black-box
• reliable
• safety-critical settings

Innovation

• Bipartite Graph
• Semantic Uncertainty
• Cross-temperature Agreement
• NLI-based Entailment
• Black-box Models