Exemplar-Guided Planing: Enhanced LLM Agent for KGQA

2025-10-16

AI Summary

To address planning inaccuracies and inefficient graph exploration in Large Language Models (LLMs) for Knowledge Graph Question Answering (KGQA)—stemming from the semantic gap between natural language queries and structured knowledge graphs—this paper proposes Exemplar-Guided Planning (EGP). EGP normalizes training data via entity templates, enables dynamic in-context example retrieval through semantic embeddings and FAISS indexing, and incorporates a Smart Lookahead mechanism to anticipate reasoning paths, refine relation pruning, and align task decomposition. Integrated into the Plan-on-Graph (PoG) framework, EGP forms the PoG-EGP system. Experiments on WebQSP and ComplexWebQuestions (CWQ) demonstrate that PoG-EGP significantly improves both question-answering accuracy and graph navigation efficiency, outperforming state-of-the-art training-free and fine-tuning baselines. Notably, PoG-EGP is the first LLM-based KGQA planner that achieves generalizable, interpretable reasoning through semantic example retrieval.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large Language Models (LLMs) as interactive agents show significant promise in Knowledge Graph Question Answering (KGQA) but often struggle with the semantic gap between natural language queries and structured knowledge graph (KG) representations. This leads to suboptimal planning and inefficient exploration on KG, while training-free approaches often underutilize valuable reasoning patterns in training data. To address these limitations, we propose a novel framework, Exemplar-Guided Planning (EGP), which enhances the planning capabilities of LLM agents for KGQA. EGP first preprocesses the training set questions via entity templating to normalize semantic variations. It then retrieves highly similar exemplary questions and their successful reasoning paths from this preprocessed set using semantic embeddings and an efficient FAISS index. These retrieved exemplars dynamically guide the LLM's planning process in two key phases: (1) Task Decomposition, by aligning generated sub-objectives with proven reasoning steps, and (2) Relation Exploration, by providing high-quality auxiliary information to improve relation pruning accuracy. Additionally, we introduce a Smart Lookahead mechanism during relation exploration to improve efficiency by preemptively exploring promising paths and potentially terminating exploration earlier. We apply EGP to the Plan-on-Graph (PoG) framework, termed PoG-EGP. Extensive experiments on two real-world KGQA datasets, WebQSP and CWQ, demonstrate that PoG-EGP significantly improves over the baseline PoG system and other compared methods.

Problem

• Bridges semantic gap between natural language and knowledge graphs
• Improves planning efficiency for knowledge graph question answering
• Enhances relation exploration accuracy through exemplar-guided reasoning

Innovation

• Uses entity templating to normalize semantic variations
• Retrieves exemplars via semantic embeddings and FAISS index
• Applies Smart Lookahead mechanism for efficient path exploration