Step-by-Step Fact Verification System for Medical Claims with Explainable Reasoning

2025-02-20

AI Summary

This work addresses the challenges of verifying factual claims and ensuring interpretability in medical domains. We propose an iterative, question-driven fact verification framework grounded in large language models (LLMs), which departs from conventional approaches relying on short evidence snippets and single-step encoder-based inference. Our method employs multi-turn self-generated contextual questioning, dynamic web retrieval, and structured logical predicate reasoning to enable stepwise, transparent, and domain-adapted verification. To our knowledge, this is the first application of iterative, stepwise verification to real-world clinical scenarios, accompanied by a novel medical fact-checking evaluation benchmark. Extensive experiments on three expert-curated medical datasets demonstrate substantial improvements over state-of-the-art methods, validating the framework’s effectiveness, interpretability, and generalizability in specialized domains.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Fact verification (FV) aims to assess the veracity of a claim based on relevant evidence. The traditional approach for automated FV includes a three-part pipeline relying on short evidence snippets and encoder-only inference models. More recent approaches leverage the multi-turn nature of LLMs to address FV as a step-by-step problem where questions inquiring additional context are generated and answered until there is enough information to make a decision. This iterative method makes the verification process rational and explainable. While these methods have been tested for encyclopedic claims, exploration on domain-specific and realistic claims is missing. In this work, we apply an iterative FV system on three medical fact-checking datasets and evaluate it with multiple settings, including different LLMs, external web search, and structured reasoning using logic predicates. We demonstrate improvements in the final performance over traditional approaches and the high potential of step-by-step FV systems for domain-specific claims.

Problem

• Develops step-by-step fact verification for medical claims.
• Enhances explainability in medical claim verification processes.
• Evaluates iterative FV systems on domain-specific datasets.

Innovation

• Iterative FV system
• Multi-turn LLMs
• Structured logic predicates