TRIAGE: Trustworthy Retrieval Instrumentation And Graph Evaluation

2026-07-03

AI Summary

This work addresses the lack of trustworthy, stage-wise evaluation in existing Graph-RAG systems, which hinders precise error localization. The authors propose TRIAGE, a novel framework that introduces the first phase-aware evaluation mechanism spanning the entire Graph-RAG pipeline. TRIAGE establishes an interpretable diagnostic chain through a three-stage metric suite: KG construction (assessing triple confidence, source coverage, and schema consistency), KG validation (measuring structural quality, correctness, and completeness), and KG utilization (evaluating retrieval coverage, faithfulness, and query cost). This enables accurate mapping of failures to specific, tunable components. The framework supports both online unsupervised diagnosis in gold-label-free settings and offline supervised calibration when ground truth is available, and is accompanied by a formal theoretical foundation, a proof-of-concept implementation, and a reproducible evaluation protocol.

Abstract

Knowledge graphs (KGs) that underpin Graph-based Retrieval-Augmented Generation (Graph-RAG) are increasingly built automatically by LLM-driven extraction rather than curated by experts. Proper evaluation would require instrumenting all pertinent stages: extraction, graph construction, and inference, coherently enough to localize failures, so that a failure at one stage is not discovered as a wrong answer at the end. We introduce TRIAGE, a stage-aware instrumentation framework for automated, document-grounded graph-RAG that asks not only whether the underlying graph can be trusted but at what cost it can be queried. TRIAGE attaches stage-specific, independently interpretable metrics to three stages: the KG Implementation (triple confidence, source coverage, and schema and canonicalization checks), the KG Validation by expert (graph-level structural quality, with correctness and completeness computed only as offline calibration when a reference is available), and the KG Usage (retrieval coverage, faithfulness, and retrieval cost); the deployed metrics need no gold annotations, the gold-requiring ones serving only as offline calibration. At usage time these metrics form a diagnostic chain of necessary conditions whose first broken link localizes the failure, and the diagnosis maps to the stage levers that can remedy it: extraction, graph and schema, or retrieval. TRIAGE is a theoretical framework with a proof of concept and a reproducible evaluation protocol.

Problem

• knowledge graph
• retrieval-augmented generation
• trustworthiness
• evaluation framework
• failure localization

Innovation

• stage-aware instrumentation
• graph-based RAG
• knowledge graph evaluation
• failure localization
• retrieval-augmented generation