When Do Supervised UQ Ensembles Improve LLM Hallucination Detection? A Robustness Study

2026-08-25

AI Summary

研究通过监督集成框架，利用少量特定领域数据集上的异构UQ评分输出训练分类器，以提高LLM在无证据情况下的幻觉检测的鲁棒性。

Abstract

Uncertainty quantification (UQ) methods are widely used for hallucination detection in large language models (LLMs) in closed-book settings where ground-truth evidence is unavailable at inference time. Prior work has proposed combining UQ signals via learned ensembles, but empirical investigations into the robustness of these ensembles are limited. We study a supervised ensembling framework that trains a classifier over heterogeneous UQ-based scorer outputs on a small, domain-specific dataset of labeled LLM responses, then applies it to out-of-sample hallucination classification without retrieval, tools, or reference documents. Across four LLMs, nine datasets, and three generation regimes (short-form QA, long-form generation, and code generation), we provide a systematic robustness analysis along three axes: sample efficiency, in-domain dataset transfer, and generation regime dependence. We find that supervised ensembles outperform the best individual scorer in 30 of 32 settings, with gains realized from as few as 100 labeled instances. Ensembles retain most of their advantage in cases of in-domain transfer under distribution shift, outperforming the best non-ensemble scorer in 23 of 28 transfer settings. Sampling-based black-box ensembles are nearly as effective as full ensembles, while single-generation white-box ensembles offer limited benefit.

Problem

• Uncertainty Quantification
• Hallucination Detection
• Large Language Models
• Robustness
• Ensemble Methods

Innovation

• Supervised Ensembling
• Uncertainty Quantification (UQ)
• Hallucination Detection
• Large Language Models (LLMs)