CORTEX: Token-Level Hallucination Detection in RAG via Comparative Internal Representations

2026-06-29

AI Summary

This work addresses the challenge of fine-grained hallucination detection in long-context retrieval-augmented generation (RAG), where localized hallucinations remain difficult to identify. The authors propose a token-level hallucination detection method that leverages internal representations of large language models (LLMs) by comparing hidden states under two conditions—with and without retrieved documents—to pinpoint tokens minimally influenced by the provided evidence. To enhance detection consistency and reduce false positives, the approach incorporates contextual information propagation modeling and sequence-level label smoothing as post-processing steps. Extensive experiments across two RAG benchmarks and three LLMs demonstrate that the proposed method significantly outperforms existing techniques, with ablation studies confirming consistent performance gains from each component.

Abstract

In this paper, we propose CORTEX, a token-level hallucination detection method for Retrieval-Augmented Generation (RAG). In long-form RAG outputs, hallucinations often arise in localized spans rather than throughout an entire response. CORTEX therefore identifies ungrounded content at the token level, enabling fine-grained localization of hallucinations. The key intuition behind CORTEX is that tokens grounded in retrieved documents should be more strongly influenced by those documents than hallucinated tokens. To capture this document-induced effect, CORTEX compares internal representations of a large language model (LLM) under two conditions: with and without the retrieved documents. Instead of relying solely on each token's immediate sensitivity to the retrieved documents, CORTEX also leverages the propagation of document-grounded information through preceding tokens, reducing false positives for tokens whose evidence has already been absorbed into the context. Finally, CORTEX applies post-processing smoothing step that models the tendency of hallucination labels to persist over contiguous spans, reducing local noise and encouraging span-consistent predictions. Experiments on two RAG benchmarks and three LLMs show that CORTEX substantially improves token-level hallucination detection, with each component consistently contributing to performance gains.

Problem

• hallucination detection
• Retrieval-Augmented Generation
• token-level
• RAG
• ungrounded content

Innovation

• hallucination detection
• token-level analysis
• retrieval-augmented generation
• internal representations
• comparative modeling