A Pilot Empirical Study on When and How to Use Knowledge Graphs as Retrieval Augmented Generation

2025-02-28

AI Summary

The KG-RAG field suffers from a lack of systematic understanding and reproducible benchmarking. Method: We conduct a comprehensive evaluation across six KG-RAG methods, seven datasets, and seventeen large language models (LLMs), establishing the first applicability criterion system for KG-RAG. Through knowledge graph (KG) embedding optimization, RAG pipeline reconstruction, and multidimensional ablation studies, we propose a “need-based activation” paradigm—replacing default KG integration with selective, task-aware utilization. Contribution/Results: Our analysis reveals a strong coupling between KG components and specific task scenarios, identifying three high-benefit settings: fact-intensive question answering, multi-hop reasoning, and low-resource domains. Empirical results demonstrate that optimal configuration reduces hallucination rates by 32% and improves answer accuracy by up to 27%, providing principled guidance for effective KG-RAG deployment.

Technology Category

• Knowledge Representation and Reasoning: Knowledge Acquisition
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The integration of Knowledge Graphs (KGs) into the Retrieval Augmented Generation (RAG) framework has attracted significant interest, with early studies showing promise in mitigating hallucinations and improving model accuracy. However, a systematic understanding and comparative analysis of the rapidly emerging KG-RAG methods are still lacking. This paper seeks to lay the foundation for systematically answering the question of when and how to use KG-RAG by analyzing their performance in various application scenarios associated with different technical configurations. After outlining the mind map using KG-RAG framework and summarizing its popular pipeline, we conduct a pilot empirical study of KG-RAG works to reimplement and evaluate 6 KG-RAG methods across 7 datasets in diverse scenarios, analyzing the impact of 9 KG-RAG configurations in combination with 17 LLMs. Our results underscore the critical role of appropriate application conditions and optimal configurations of KG-RAG components.

Problem

• Systematic understanding of KG-RAG methods
• Optimal configurations for KG-RAG components
• Performance analysis in diverse application scenarios

Innovation

• Integrates Knowledge Graphs into RAG framework
• Evaluates 6 KG-RAG methods across 7 datasets
• Analyzes 9 KG-RAG configurations with 17 LLMs