Mitigating Prompt-Induced Hallucinations in Large Language Models via Structured Reasoning

2026-01-06

arXiv.org

AI Summary

This work addresses the challenge of prompt-induced hallucinations in large language models by proposing a structured reasoning approach that integrates code-guided exploration with knowledge graphs. The method embeds executable code as a structured knowledge carrier within chain-of-thought prompting, enabling the model to perform controllable traversal over a knowledge graph. A chained knowledge distillation architecture further refines the reasoning process. This approach achieves the first implementation of code-driven external knowledge injection, significantly enhancing factual accuracy on both GPT-4 and LLaMA-3.3. Experimental results demonstrate consistent improvements across multiple settings, with HIT@1, HIT@3, and HIT@5 scores increasing by 15.64%, 13.38%, and 13.28%, respectively, and overall hit rates exceeding 95% in various evaluation scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

To address hallucination issues in large language models (LLMs), this paper proposes a method for mitigating prompt-induced hallucinations. Building on a knowledge distillation chain-style model, we introduce a code module to guide knowledge-graph exploration and incorporate code as part of the chain-of-thought prompt, forming an external knowledge input that provides more accurate and structured information to the model. Based on this design, we develop an improved knowledge distillation chain-style model and leverage it to analyze and constrain the reasoning process of LLMs, thereby improving inference accuracy. We empirically evaluate the proposed approach using GPT-4 and LLaMA-3.3 on multiple public datasets. Experimental results demonstrate that incorporating code modules significantly enhances the model's ability to capture contextual information and effectively mitigates prompt-induced hallucinations. Specifically, HIT@1, HIT@3, and HIT@5 improve by 15.64%, 13.38%, and 13.28%, respectively. Moreover, the proposed method achieves HIT@1, HIT@3, and HIT@5 scores exceeding 95% across several evaluation settings. These results indicate that the proposed approach substantially reduces hallucination behavior while improving the accuracy and verifiability of large language models.

Problem

• hallucination
• large language models
• prompt-induced
• structured reasoning
• knowledge distillation

Innovation

• structured reasoning
• prompt-induced hallucinations
• code-augmented chain-of-thought
• knowledge distillation
• knowledge graph exploration