Detecting Hallucinations in Retrieval-Augmented Generation through Grounding-Aware Sensitivity by Perturbation (GASP)

2026-07-05

AI Summary

This work addresses the limitation of existing retrieval-augmented generation (RAG) systems, which lack fine-grained hallucination localization and can only provide coarse-grained answer-level confidence scores. The authors propose GASP, a novel training-free method that introduces sentence-level “grounding sensitivity” to quantify how much each generated sentence relies on retrieved evidence. By perturbing the retrieval context while keeping the generated output fixed, GASP computes changes in log-likelihood and Jensen–Shannon divergence across contexts to assess grounding fidelity. The approach models decoding as a stochastic nonlinear iterated function system and integrates context ablation, likelihood differencing, and a training-free thresholding mechanism for efficient hallucination detection. Evaluated on the RAGTruth benchmark, GASP achieves response-level and span-level AUCs of 0.73 and 0.67, respectively, substantially outperforming baselines based on perplexity, natural language inference, and self-consistency.

Abstract

Retrieval-augmented generation (RAG) reduces but does not eliminate hallucination, and existing detectors return a single answer-level score that does not indicate which sentence is unsupported, or why. To close this gap, we introduce Grounding-Aware Sensitivity by Perturbation (GASP), a span-level detector that scores each answer sentence by how strongly its likelihood depends on the retrieved evidence, a quantity we term grounding sensitivity. GASP holds the answer fixed and re-scores it under the full context, under no context, and with each chunk removed, then measures the log-likelihood drops and Jensen-Shannon divergences (JSD). The likelihood of a grounded sentence collapses once its supporting passage is removed, whereas a hallucinated sentence is almost unaffected, a contrast we interpret by casting decoding as a random nonlinear iterated function system (RNIFS). We evaluate GASP on three benchmarks (RAGTruth, TofuEval, RAGBench) with three instruction-tuned scorers from two model families (Qwen2.5-0.5B, Qwen2.5-1.5B, and SmolLM2-1.7B) under a leakage-clean protocol. On RAGTruth it reaches a response-level area under the ROC curve (AUC) of about 0.73 and a span-level AUC of about 0.67, improving significantly over perplexity and by clear margins over length, whole-context natural language inference (NLI), and self-consistency baselines. The only baseline competitive at the span level is a well-configured chunk-level entailment verifier, which requires a separate model, whereas a training-free threshold on the grounding features matches the trained classifier without labeled data and serves as the default detector. Beyond RAGTruth, the signal transfers to TofuEval but not to short-answer question answering in RAGBench, showing GASP is best suited to outputs constructed from the retrieved context rather than answers recoverable from parametric knowledge.

Problem

• hallucination detection
• retrieval-augmented generation
• span-level evaluation
• grounding
• evidence support

Innovation

• Grounding-Aware Sensitivity
• Retrieval-Augmented Generation
• Hallucination Detection
• Span-Level Evaluation
• Perturbation-Based Scoring