LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations

2024-10-03

International Conference on Learning Representations

AI Summary

Large language models (LLMs) frequently generate hallucinations—including factual inaccuracies, biases, and reasoning failures—despite encoding truth-relevant information in their internal representations. However, the structural organization of this veracity information remains poorly understood, and existing hallucination detection methods suffer from limited generalizability, suggesting that truth encoding is multifaceted rather than mediated by a single “truth channel.” Method: We conduct systematic analysis of hidden-layer activations and perform token-level attribution to localize fine-grained hallucination representations within LLMs. Contribution/Results: We find that veracity signals are highly concentrated at specific tokens; refute the “universal truth-encoding” hypothesis and propose a multifaceted encoding perspective; achieve, for the first time, hallucination-type prediction and implicit correct-answer identification; and uncover a cognitive inconsistency wherein models internally “know” the correct answer yet externally produce erroneous outputs. Our method significantly improves hallucination detection across diverse datasets, establishing a novel paradigm for hallucination analysis and mitigation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) often produce errors, including factual inaccuracies, biases, and reasoning failures, collectively referred to as"hallucinations". Recent studies have demonstrated that LLMs' internal states encode information regarding the truthfulness of their outputs, and that this information can be utilized to detect errors. In this work, we show that the internal representations of LLMs encode much more information about truthfulness than previously recognized. We first discover that the truthfulness information is concentrated in specific tokens, and leveraging this property significantly enhances error detection performance. Yet, we show that such error detectors fail to generalize across datasets, implying that -- contrary to prior claims -- truthfulness encoding is not universal but rather multifaceted. Next, we show that internal representations can also be used for predicting the types of errors the model is likely to make, facilitating the development of tailored mitigation strategies. Lastly, we reveal a discrepancy between LLMs' internal encoding and external behavior: they may encode the correct answer, yet consistently generate an incorrect one. Taken together, these insights deepen our understanding of LLM errors from the model's internal perspective, which can guide future research on enhancing error analysis and mitigation.

Problem

• LLMs encode hidden truthfulness information in internal states
• Error detection performance improves by focusing on specific tokens
• LLMs' internal truthfulness encoding is multifaceted, not universal

Innovation

• Detect errors using LLMs' internal truthfulness information
• Predict error types from internal representations for mitigation
• Reveal discrepancy between internal encoding and external behavior