Ex-GraphRAG: Interpretable Evidence Routing for Graph-Augmented LLMs

2026-05-21

AI Summary

Existing graph-augmented large language models suffer from limited auditability due to the use of conventional graph neural networks (GNNs) for subgraph encoding, which entangles node contributions and obscures the critical evidence influencing model outputs. To address this, this work proposes the Multivariate Graph Neural Additive Network (M-GNAN), the first approach enabling exact, post-hoc approximation-free interpretability in graph-augmented LLMs by decomposing encoder outputs precisely along both nodes and feature groups. Evaluated on STaRK-Prime, M-GNAN matches the performance of black-box GNNs while revealing a key mismatch: semantic importance and structural connectivity are often governed by distinct node sets. Auditing further demonstrates that omitting crucial intermediate nodes can degrade multi-hop reasoning performance by up to 28%, offering new insights for retrieval pruning and context construction.

Abstract

GraphRAG conditions language models on subgraphs retrieved from knowledge graphs, encoded via message-passing GNNs. Because these encoders entangle node contributions through iterated neighborhood aggregation, there is no closed-form way to determine how much each retrieved entity influenced the encoder's output, and therefore no way to faithfully audit what structural evidence actually reached the model. We introduce Ex-GraphRAG, which replaces the GNN encoder with a Multivariate Graph Neural Additive Network (M-GNAN), an extension of additive graph models to high-dimensional embedding spaces that yields an exact decomposition of the encoder's output across individual nodes and feature groups, without post-hoc approximation. On STaRK-Prime, this auditable encoder matches black-box performance. Using it to audit evidence routing, we uncover a semantic-structural mismatch: the nodes that dominate the encoder's output are structurally disconnected in the retrieved subgraph, held together by low-attribution intermediaries whose removal degrades multi-hop QA by up to 28%. This mismatch, invisible to any opaque encoder, reveals that semantic importance and structural connectivity are governed by disjoint sets of nodes, with direct implications for retrieval pruning, context construction, and failure diagnosis in graph-augmented LLMs.

Problem

• GraphRAG
• interpretability
• evidence routing
• knowledge graphs
• large language models

Innovation

• Explainable AI
• Graph Neural Additive Networks
• Evidence Routing
• Graph-Augmented LLMs
• Interpretability