KARPA: A Training-free Method of Adapting Knowledge Graph as References for Large Language Model's Reasoning Path Aggregation

2024-12-30

AI Summary

Large language models (LLMs) suffer from hallucination, knowledge staleness, reliance on local graph traversal, and dependency on task-specific fine-tuning or retrieval-augmented generation (RAG) in knowledge graph question answering (KGQA). Method: We propose a training-free, fine-tuning-free reasoning path aggregation framework that leverages LLMs’ global reasoning capability to pre-generate multiple semantically plausible relation paths over the KG; answers are then produced zero-shot via path embedding matching and multi-path aggregation. Contribution/Results: This work is the first to harness LLMs’ global planning ability for KG path pre-generation and aggregation; it supports plug-and-play integration with arbitrary LLMs and abandons conventional stepwise retrieval paradigms. Evaluated on mainstream KGQA benchmarks, our method achieves state-of-the-art accuracy—significantly outperforming both fine-tuned and RAG-based approaches—while improving inference efficiency by over 40% and incurring zero training overhead.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Knowledge Representation and Reasoning: Knowledge Acquisition

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) demonstrate exceptional performance across a variety of tasks, yet they are often affected by hallucinations and the timeliness of knowledge. Leveraging knowledge graphs (KGs) as external knowledge sources has emerged as a viable solution, but existing methods for LLM-based knowledge graph question answering (KGQA) are often limited by step-by-step decision-making on KGs, restricting the global planning and reasoning capabilities of LLMs, or they require fine-tuning or pre-training on specific KGs. To address these challenges, we propose Knowledge graph Assisted Reasoning Path Aggregation (KARPA), a novel framework that harnesses the global planning abilities of LLMs for efficient and accurate KG reasoning. KARPA operates in three steps: pre-planning relation paths using the LLM's global planning capabilities, matching semantically relevant paths via an embedding model, and reasoning over these paths to generate answers. Unlike existing KGQA methods, KARPA avoids stepwise traversal, requires no additional training, and is adaptable to various LLM architectures. Extensive experimental results show that KARPA achieves state-of-the-art performance in KGQA tasks, delivering both high efficiency and accuracy. Our code will be available on Github.

Problem

• Large Language Models
• Knowledge Graph Utilization
• Accuracy and Efficiency

Innovation

• KARPA
• Large Language Models
• Knowledge Graph Querying