🤖 AI Summary
This work addresses the challenge of hallucinations—factual inaccuracies in text generated by large language models—and proposes CHAI R, a novel decoding-time detection framework. CHAI R is the first to systematically leverage statistical features derived from multi-layer internal logits, such as extrema, mean, standard deviation, and slope, to construct compact representations for hallucination identification via supervised learning. Evaluated on TruthfulQA and MMLU benchmarks, the method significantly improves detection accuracy and demonstrates strong generalization capabilities, particularly in zero-shot settings. By exploiting internal model representations, this approach offers a promising direction for adaptive decoding strategies aimed at enhancing the factual consistency of language model outputs.
📝 Abstract
In this work, we introduce CHAIR (Classifier of Hallucination As ImproveR), a supervised framework for detecting hallucinations by analyzing internal logits from each layer of every token. Our method extracts a compact set of features such as maximum, minimum, mean, standard deviation, and slope-from the token logits across all layers, enabling effective hallucination detection without overfitting. Experiments on TruthfulQA and MMLU datasets demonstrate that CHAIR significantly improves detection accuracy, particularly in zero-shot scenarios, showcasing its robustness and generalizability. Beyond hallucination detection, CHAIR highlights the potential of using internal representations for designing advanced decoding strategies. By leveraging patterns in logits, we suggest that more sophisticated models and adaptive decoding methods could further reduce hallucinations and enhance text completion quality. CHAIR not only offers a practical solution for detecting hallucinations but also lays the groundwork for exploring richer representations in LLMs to improve their factuality and coherence.