From Documents to Dialogue: Building KG-RAG Enhanced AI Assistants

2025-02-21

AI Summary

To address inaccurate zero-shot responses from large language models (LLMs) caused by restricted access to enterprise-private documents, this paper proposes KG-RAG, a knowledge graph–enhanced AI assistant framework. The core methodological innovation lies in a novel paradigm for constructing high-quality, low-noise knowledge graphs: (i) seed-concept–guided incremental entity disambiguation; (ii) semantic-similarity–driven deduplication; (iii) confidence-weighted fact filtering; and (iv) traceable document-fact alignment. Integrating entity-relation extraction, confidence modeling, and semantic retrieval, KG-RAG achieves precise, interpretable zero-shot question answering without fine-tuning or external data. Experiments demonstrate that, compared to existing production systems, KG-RAG reduces irrelevant answers by 52% and increases fully relevant answers by 88%, significantly improving both response accuracy and provenance traceability.

Technology Category

• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Natural Language Processing: Question Answering
• Knowledge Representation and Reasoning: Knowledge Acquisition

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The Adobe Experience Platform AI Assistant is a conversational tool that enables organizations to interact seamlessly with proprietary enterprise data through a chatbot. However, due to access restrictions, Large Language Models (LLMs) cannot retrieve these internal documents, limiting their ability to generate accurate zero-shot responses. To overcome this limitation, we use a Retrieval-Augmented Generation (RAG) framework powered by a Knowledge Graph (KG) to retrieve relevant information from external knowledge sources, enabling LLMs to answer questions over private or previously unseen document collections. In this paper, we propose a novel approach for building a high-quality, low-noise KG. We apply several techniques, including incremental entity resolution using seed concepts, similarity-based filtering to deduplicate entries, assigning confidence scores to entity-relation pairs to filter for high-confidence pairs, and linking facts to source documents for provenance. Our KG-RAG system retrieves relevant tuples, which are added to the user prompts context before being sent to the LLM generating the response. Our evaluation demonstrates that this approach significantly enhances response relevance, reducing irrelevant answers by over 50% and increasing fully relevant answers by 88% compared to the existing production system.

Problem

• Enhancing AI assistant with KG-RAG
• Overcoming LLM access restrictions
• Improving response relevance with KG

Innovation

• KG-RAG framework
• incremental entity resolution
• confidence scores filtering