Multi-stage Prompt Refinement for Mitigating Hallucinations in Large Language Models

2025-10-13

AI Summary

Large language models (LLMs) frequently exhibit hallucinations due to ill-formed prompts—characterized by grammatical errors, ambiguous wording, or missing information—yet this causal relationship remains systematically unexplored. Method: We propose a multi-stage prompt optimization framework that employs lightweight fine-tuned small models to iteratively correct punctuation, typos, and keywords, augmented with context expansion and a ranking-based self-reflection mechanism to enhance prompt clarity and structural coherence. Contribution/Results: This is the first work to systematically analyze how prompt defects induce hallucinations and to realize iterative prompt purification. The framework is both interpretable and compatible, enabling seamless integration with existing post-hoc mitigation methods. Evaluated across multiple hallucination benchmarks, prompts optimized by our method achieve an 85.3% win rate over original prompts, significantly improving LLM output accuracy and robustness.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advancements in large language models (LLMs) have shown strong performance in natural language understanding and generation tasks. However, LLMs continue to encounter challenges with hallucinations, where models generate plausible but incorrect information. While several factors contribute to hallucinations, the impact of ill-formed prompts, prompts with ambiguous wording, incorrect grammar, or incomplete information, was relatively under explored. To address this, we introduce Multi-stage Prompt Refinement (MPR), a framework designed to systematically improve these ill-formed prompts across multiple stages. Each stage addresses specific errors such as punctuation, typographical mistakes, and misuse of key terms, using small language models (SLMs) fine-tuned for these tasks. MPR iteratively enhances the clarity of prompts with additional context and employs a self-reflection mechanism with ranking to prioritize the most relevant input. Experimental results on hallucination benchmarks show that prompts refined by MPR achieve over an 85~% win rate compared to their original forms, demonstrating its effectiveness in reducing hallucinations and improving LLM output accuracy. Interestingly, we reveal that MPR can be combined with existing post-hoc hallucination mitigation frameworks, further enhancing its versatility. MPR provides a lightweight and adaptable solution for enhancing LLM reliability across various domains.

Problem

• Reducing hallucinations in large language models caused by ill-formed prompts
• Improving prompt clarity through multi-stage refinement of grammatical errors
• Enhancing LLM output accuracy with lightweight iterative prompt optimization

Innovation

• Multi-stage Prompt Refinement framework improves prompts
• Fine-tuned small language models correct specific errors
• Self-reflection mechanism ranks inputs for relevance