Bolster Hallucination Detection via Prompt-Guided Data Augmentation

2025-10-12

AI Summary

To address the scarcity of high-quality labeled data for hallucination detection in large language models (LLMs), this paper proposes a fully unsupervised, prompt-guided hallucination detection framework. Methodologically, it innovatively integrates prompt engineering with data augmentation to automatically generate paired truthful and hallucinated responses; subsequently, hidden-layer activation representations are extracted via matrix decomposition, and a contrastive Mahalanobis distance is introduced to quantify distributional divergence between the two response types in the embedding space—enabling end-to-end modeling. The core contribution lies in the first integration of prompt-driven synthetic data generation with activation-space-based contrastive distribution modeling, thereby eliminating reliance on human annotations. Experiments demonstrate that the method achieves an average 6.55% improvement over strong baselines across multiple benchmarks, significantly enhancing both detection accuracy and cross-model generalization capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) have garnered significant interest in AI community. Despite their impressive generation capabilities, they have been found to produce misleading or fabricated information, a phenomenon known as hallucinations. Consequently, hallucination detection has become critical to ensure the reliability of LLM-generated content. One primary challenge in hallucination detection is the scarcity of well-labeled datasets containing both truthful and hallucinated outputs. To address this issue, we introduce Prompt-guided data Augmented haLlucination dEtection (PALE), a novel framework that leverages prompt-guided responses from LLMs as data augmentation for hallucination detection. This strategy can generate both truthful and hallucinated data under prompt guidance at a relatively low cost. To more effectively evaluate the truthfulness of the sparse intermediate embeddings produced by LLMs, we introduce an estimation metric called the Contrastive Mahalanobis Score (CM Score). This score is based on modeling the distributions of truthful and hallucinated data in the activation space. CM Score employs a matrix decomposition approach to more accurately capture the underlying structure of these distributions. Importantly, our framework does not require additional human annotations, offering strong generalizability and practicality for real-world applications. Extensive experiments demonstrate that PALE achieves superior hallucination detection performance, outperforming the competitive baseline by a significant margin of 6.55%.

Problem

• Detecting hallucinations in large language model outputs
• Addressing scarcity of labeled hallucination datasets
• Evaluating truthfulness through contrastive embedding analysis

Innovation

• Prompt-guided data augmentation for hallucination detection
• Contrastive Mahalanobis Score evaluates sparse embeddings
• Framework requires no human annotations for practicality