Detecting Hallucinations in Graph Retrieval-Augmented Generation via Attention Patterns and Semantic Alignment

2025-12-09

AI Summary

In GraphRAG, large language models (LLMs) struggle to interpret the topological structure of knowledge graph subgraphs, leading to hallucinations inconsistent with retrieved knowledge. To address this, we propose GGA—a mechanism-based, post-hoc hallucination detector grounded in interpretability. GGA introduces two lightweight, interpretable metrics: Path Reliance Degree (PRD), quantifying model dependence on graph paths, and Semantic Alignment Score (SAS), measuring alignment between generated outputs and retrieved subgraph semantics. It further integrates attention-pattern analysis with semantic representation modeling to precisely localize hallucinations. Experiments demonstrate that GGA significantly outperforms semantic- and confidence-based baselines in AUC and F1. Empirical analysis identifies “high PRD + low SAS” as a robust discriminative signal for hallucination. This work is the first to incorporate mechanistic interpretability into hallucination detection for GraphRAG, establishing a novel paradigm for trustworthy, graph-augmented generation.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Graph-based Retrieval-Augmented Generation (GraphRAG) enhances Large Language Models (LLMs) by incorporating external knowledge from linearized subgraphs retrieved from knowledge graphs. However, LLMs struggle to interpret the relational and topological information in these inputs, resulting in hallucinations that are inconsistent with the retrieved knowledge. To analyze how LLMs attend to and retain structured knowledge during generation, we propose two lightweight interpretability metrics: Path Reliance Degree (PRD), which measures over-reliance on shortest-path triples, and Semantic Alignment Score (SAS), which assesses how well the model's internal representations align with the retrieved knowledge. Through empirical analysis on a knowledge-based QA task, we identify failure patterns associated with over-reliance on salient paths and weak semantic grounding, as indicated by high PRD and low SAS scores. We further develop a lightweight post-hoc hallucination detector, Graph Grounding and Alignment (GGA), which outperforms strong semantic and confidence-based baselines across AUC and F1. By grounding hallucination analysis in mechanistic interpretability, our work offers insights into how structural limitations in LLMs contribute to hallucinations, informing the design of more reliable GraphRAG systems in the future.

Problem

• Detect hallucinations in GraphRAG systems caused by LLMs misinterpreting graph structures.
• Measure model over-reliance on path triples and weak semantic alignment with knowledge.
• Develop interpretable metrics and detectors to improve GraphRAG reliability and reduce errors.

Innovation

• Proposes Path Reliance Degree to measure over-reliance on triples
• Introduces Semantic Alignment Score to assess knowledge representation alignment
• Develops Graph Grounding and Alignment detector for post-hoc hallucination detection