Chatting with Papers: A Hybrid Approach Using LLMs and Knowledge Graphs

2025-05-16

AI Summary

This study addresses key challenges in navigating academic paper collections—namely, difficulty in obtaining overviews, shallow conceptual understanding, and ambiguous research problem formulation. To this end, we propose a retrieval-augmented generation (RAG) framework that synergistically integrates large language models (LLMs) with dynamic knowledge graphs. Methodologically, we develop a locally adaptable, interactive academic chat system: semantic knowledge graphs are automatically constructed from paper metadata and full texts; context-aware retrieval, explanation, and iterative question refinement are achieved through joint graph querying and LLM inference. Our key contribution is the first integration of dynamic knowledge graphs into the LLM dialogue flow, enabling semantic navigation, concept provenance tracing, and controllable problem evolution. Evaluation on the GESI methodology journal corpus demonstrates significant improvements: average exploration time reduced by 37%, and expert-assessed research problem definition accuracy increased by 29%.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Knowledge Representation and Reasoning: Knowledge Acquisition
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

This demo paper reports on a new workflow extit{GhostWriter} that combines the use of Large Language Models and Knowledge Graphs (semantic artifacts) to support navigation through collections. Situated in the research area of Retrieval Augmented Generation, this specific workflow details the creation of local and adaptable chatbots. Based on the tool-suite extit{EverythingData} at the backend, extit{GhostWriter} provides an interface that enables querying and ``chatting'' with a collection. Applied iteratively, the workflow supports the information needs of researchers when interacting with a collection of papers, whether it be to gain an overview, to learn more about a specific concept and its context, and helps the researcher ultimately to refine their research question in a controlled way. We demonstrate the workflow for a collection of articles from the extit{method data analysis} journal published by GESIS -- Leibniz-Institute for the Social Sciences. We also point to further application areas.

Problem

• Combining LLMs and Knowledge Graphs for collection navigation
• Creating local adaptable chatbots for research papers
• Supporting researchers in refining questions via paper interaction

Innovation

• Combines LLMs and Knowledge Graphs for navigation
• Creates local, adaptable chatbots for collections
• Supports querying and chatting with paper collections