KGCaRe: Explainable Complex Conditional Question Answering using Automatic Knowledge Graph Construction and Context Retrieval with LLMs

2026-08-10

AI Summary

This work addresses the challenge that large language models (LLMs) and conventional retrieval-augmented generation approaches struggle to accurately answer complex conditional questions in domain-specific settings. The authors propose a hybrid architecture integrating neural retrieval with symbolic reasoning, which automatically constructs knowledge graphs from documents via a multi-prompt strategy and stores them in a graph database while simultaneously building a vector index for semantic retrieval. They introduce an LLM-guided iterative graph traversal mechanism coupled with a clue-entity re-traversal strategy to jointly retrieve textual passages and graph triple paths, thereby generating answers that are both interpretable and highly accurate. Experiments demonstrate that the method significantly outperforms baselines—including vanilla LLMs, Think-on-Graph, and HybridContextQA—across two complex conditional question answering benchmarks, with consistent gains observed using diverse LLMs such as Mistral, Mixtral, GPT-3.5, and GPT-4o. The complete software pipeline has been open-sourced.

Abstract

Answering complex conditional questions using Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) remains a challenge, particularly in domain-specific contexts where general-purpose LLMs and RAG tend to underperform. We hypothesize that augmenting RAG with unstructured and structured knowledge, extracted from both documents and knowledge graphs (KGs), can improve reasoning and answer accuracy for such tasks. To test this, we propose KGCaRe, a hybrid approach that combines neural retrieval with symbolic reasoning over LLM-generated KGs. KGCaRe constructs a KG from documents using a multi-prompt extraction strategy and stores it in a graph database. Simultaneously, the documents are embedded into a vector store to enable neural retrieval. KGCaRe performs innovative iterative graph traversal guided by the LLM to extract relevant triples, prune irrelevant information, and uses additional clue entities to traverse the graph again if the initial traversal does not provide satisfactory context to generate the answer. The relevant triples extracted from the KG in path form, along with semantically retrieved text passages, are then fed into custom KGCaRe prompts to generate answers to the complex conditional questions with explanations. We evaluate KGCaRe on two complex conditional QA datasets. Our results on these datasets show that KGCaRe consistently outperforms existing baselines, including Vanilla LLM, Code Prompt, Text Prompt, Think-on-Graph, Vanilla RAG, and HybridContextQA, across multiple LLMs such as Mistral, Mixtral, GPT-3.5, and GPT-4o. We publicly release the software pipeline that we developed to implement the proposed KGCaRe approach.

Problem

• complex conditional question answering
• domain-specific QA
• retrieval-augmented generation
• large language models
• knowledge graphs

Innovation

• Knowledge Graph Construction
• Retrieval-Augmented Generation
• Iterative Graph Traversal
• Explainable QA
• Hybrid Reasoning