Pre-Generation Hallucination Detection in Large Language Models via Soft-Target Attention Probing

2026-06-20

AI Summary

This work proposes a pre-generation hallucination risk detection method designed to support decisions such as answer refusal, retrieval augmentation, or query routing. The core innovation lies in formulating hallucination detection as a risk estimation problem: unbiased minimum-variance estimates of per-prompt hallucination probability are derived from empirical error rates obtained via stochastic sampling of model generations, serving as soft-target supervision signals. For the first time in this setting, attention probes are introduced to selectively aggregate prompt representations most relevant to hallucination. Experimental results across three question-answering benchmarks and five large language models demonstrate that attention probes consistently outperform linear probes on short-answer tasks, and that soft-target supervision reliably enhances detection performance.

Abstract

Detecting hallucination risk before generation enables abstention, retrieval augmentation, and routing decisions without incurring the cost of decoding. While prior work has shown that such risk can be estimated from a model's internal representations, existing approaches treat this as binary classification over a single decoded output. We instead formulate it as a risk-estimation problem. Under this formulation, we introduce soft-target supervision based on the empirical answer error rate over stochastically sampled outputs - an estimator we prove to be the unique unbiased minimum-variance estimator of the model's per-prompt error probability under its sampling distribution. We further adapt attention probing to the pre-generation setting, enabling the detector to selectively aggregate hallucination-relevant prompt representations. Across three question-answering benchmarks and five models, attention probing outperforms linear probing on short-answer tasks. Replacing binary labels with soft-target supervision further and consistently improves detection quality.

Problem

• hallucination detection
• large language models
• pre-generation
• risk estimation
• soft-target supervision

Innovation

• soft-target supervision
• attention probing
• pre-generation hallucination detection
• risk estimation
• unbiased minimum-variance estimator