Can Your Uncertainty Scores Detect Hallucinated Entity?

2025-02-17

AI Summary

Current large language models (LLMs) frequently generate hallucinated content, yet mainstream uncertainty estimation methods operate only at the sentence or paragraph level, failing to precisely localize hallucinated entities. To address this, we propose the first entity-level hallucination detection paradigm and introduce HalluEntity—the first fine-grained, human-annotated dataset for entity-level hallucination, accompanied by a dedicated evaluation benchmark. We systematically evaluate 17 state-of-the-art LLMs using diverse uncertainty metrics—including token-level entropy, confidence scores, logit differences, and context-aware scoring functions. Our analysis reveals that token-level methods suffer from systematic over-detection, whereas context-aware approaches achieve superior performance but remain suboptimal. Furthermore, we uncover statistically significant correlations between hallucination occurrence and linguistic attributes (e.g., entity type, syntactic role, and discourse position). This work establishes a new benchmark for precise, interpretable hallucination localization, delivers novel empirical insights into hallucination mechanisms, and opens avenues for entity-aware uncertainty modeling.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

To mitigate the impact of hallucination nature of LLMs, many studies propose detecting hallucinated generation through uncertainty estimation. However, these approaches predominantly operate at the sentence or paragraph level, failing to pinpoint specific spans or entities responsible for hallucinated content. This lack of granularity is especially problematic for long-form outputs that mix accurate and fabricated information. To address this limitation, we explore entity-level hallucination detection. We propose a new data set, HalluEntity, which annotates hallucination at the entity level. Based on the dataset, we comprehensively evaluate uncertainty-based hallucination detection approaches across 17 modern LLMs. Our experimental results show that uncertainty estimation approaches focusing on individual token probabilities tend to over-predict hallucinations, while context-aware methods show better but still suboptimal performance. Through an in-depth qualitative study, we identify relationships between hallucination tendencies and linguistic properties and highlight important directions for future research.

Problem

• Detect hallucinated entities in LLMs
• Evaluate entity-level hallucination detection
• Improve uncertainty-based detection methods

Innovation

• entity-level hallucination detection
• uncertainty estimation approaches
• context-aware methods