Don't Let It Hallucinate: Premise Verification via Retrieval-Augmented Logical Reasoning

2025-04-08

AI Summary

To address hallucination in large language models (LLMs) triggered by user queries containing false premises, this paper proposes a *preemptive verification framework*. First, the query is formalized into logical premises; second, each premise is independently verified for factual correctness via retrieval-augmented generation (RAG); finally, verification outcomes are injected into the prompt to guide factually consistent generation. Our approach introduces the novel concept of *premise-level proactive verification*—a lightweight, inference-time intervention that requires no access to model logits, fine-tuning, or additional training. It relies solely on RAG and prompt engineering to intercept hallucinations *before* generation. Experiments demonstrate significant reductions in hallucination rates and substantial improvements in factual accuracy, while maintaining low latency, robustness across diverse domains, and broad applicability to real-time inference scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large language models (LLMs) have shown substantial capacity for generating fluent, contextually appropriate responses. However, they can produce hallucinated outputs, especially when a user query includes one or more false premises-claims that contradict established facts. Such premises can mislead LLMs into offering fabricated or misleading details. Existing approaches include pretraining, fine-tuning, and inference-time techniques that often rely on access to logits or address hallucinations after they occur. These methods tend to be computationally expensive, require extensive training data, or lack proactive mechanisms to prevent hallucination before generation, limiting their efficiency in real-time applications. We propose a retrieval-based framework that identifies and addresses false premises before generation. Our method first transforms a user's query into a logical representation, then applies retrieval-augmented generation (RAG) to assess the validity of each premise using factual sources. Finally, we incorporate the verification results into the LLM's prompt to maintain factual consistency in the final output. Experiments show that this approach effectively reduces hallucinations, improves factual accuracy, and does not require access to model logits or large-scale fine-tuning.

Problem

• Detects false premises in LLM queries to prevent hallucinations
• Uses retrieval-augmented logic for real-time premise verification
• Improves factual accuracy without logits access or fine-tuning

Innovation

• Retrieval-based framework prevents false premises
• Logical representation and RAG validate premises
• Verification results enhance prompt factual consistency