Deliberation on Priors: Trustworthy Reasoning of Large Language Models on Knowledge Graphs

2025-05-21

AI Summary

To address hallucination in large language models (LLMs) during knowledge graph (KG)-enhanced retrieval-augmented generation—caused by insufficient exploitation of prior knowledge—this paper proposes the *Distrust-and-Prune* (DP) framework for trustworthy reasoning. Methodologically, DP introduces a novel dual-path prior injection mechanism integrating *structural priors* (e.g., KG topology) and *constraint priors* (explicit and implicit logical constraints), guided by Kahneman–Tversky-inspired optimization for progressive knowledge distillation. It further establishes a reasoning–introspection co-verification paradigm, supported by KG structural encoding, constraint extraction, and reflective path validation. Evaluated on ComplexWebQuestions, DP achieves a 13% absolute improvement in Hit@1, setting a new state-of-the-art. It significantly enhances relational path faithfulness and response reliability while demonstrating strong generalization and practical applicability.

Technology Category

• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Knowledge Representation and Reasoning: Reasoning with Beliefs
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Knowledge graph-based retrieval-augmented generation seeks to mitigate hallucinations in Large Language Models (LLMs) caused by insufficient or outdated knowledge. However, existing methods often fail to fully exploit the prior knowledge embedded in knowledge graphs (KGs), particularly their structural information and explicit or implicit constraints. The former can enhance the faithfulness of LLMs' reasoning, while the latter can improve the reliability of response generation. Motivated by these, we propose a trustworthy reasoning framework, termed Deliberation over Priors (DP), which sufficiently utilizes the priors contained in KGs. Specifically, DP adopts a progressive knowledge distillation strategy that integrates structural priors into LLMs through a combination of supervised fine-tuning and Kahneman-Tversky optimization, thereby improving the faithfulness of relation path generation. Furthermore, our framework employs a reasoning-introspection strategy, which guides LLMs to perform refined reasoning verification based on extracted constraint priors, ensuring the reliability of response generation. Extensive experiments on three benchmark datasets demonstrate that DP achieves new state-of-the-art performance, especially a Hit@1 improvement of 13% on the ComplexWebQuestions dataset, and generates highly trustworthy responses. We also conduct various analyses to verify its flexibility and practicality. The code is available at https://github.com/reml-group/Deliberation-on-Priors.

Problem

• Mitigate LLM hallucinations via knowledge graph priors
• Enhance reasoning faithfulness with structural KG information
• Improve response reliability through constraint-based verification

Innovation

• Progressive knowledge distillation with KG structural priors
• Kahneman-Tversky optimization for faithful relation paths
• Reasoning-introspection with constraint priors for reliability