🤖 AI Summary
To address hallucination and knowledge obsolescence in large language models (LLMs) for multi-document question answering (MD-QA), this paper proposes a fine-tuning-free, curiosity-driven reasoning mechanism. It generates semantically coherent follow-up questions to guide efficient, multi-hop traversal of knowledge graphs and evidence retrieval, augmented by an evidence-sufficiency discriminator with a dedicated termination token for dynamic question chaining. The core contributions are: (1) the first follow-up question–guided QA dataset (Follow-upQA), where follow-up questions serve as explicit supervision signals; and (2) an end-to-end prompt-based reasoning framework integrating knowledge graph navigation, adaptive follow-up generation, and multi-hop inference. Experiments demonstrate substantial accuracy gains over baselines such as KGP, while avoiding costly large-scale fine-tuning and high inference latency—achieving over 40% improvement in reasoning efficiency.
📝 Abstract
Large Language Models (LLMs) have achieved significant success in open-domain question answering. However, they continue to face challenges such as hallucinations and knowledge cutoffs. These issues can be mitigated through in-context learning by providing LLMs with relevant context before generating answers. Recent literature proposes Knowledge Graph Prompting (KGP) which integrates knowledge graphs with an LLM-based traversal agent to substantially enhance document retrieval quality. However, KGP requires costly fine-tuning with large datasets and remains prone to hallucination. In this paper, we propose CuriousLLM, an enhancement that integrates a curiosity-driven reasoning mechanism into an LLM agent. This mechanism enables the agent to generate relevant follow-up questions, thereby guiding the information retrieval process more efficiently. Central to our approach is the development of the new Follow-upQA dataset, which includes questions and supporting evidence as input, with follow-up questions serving as ground truths. These follow-up questions either inquire about what is still missing to fully answer the user's query or use special tokens to signify that the retrieved evidence is sufficient. Our experiments show that CuriousLLM significantly boosts LLM performance in multi-document question answering (MD-QA), circumventing the substantial computational costs and latency from the original KGP framework.