Rethinking LLM Verification: Evidence Structure, Uncertainty, and Selective Refinement

2026-08-11

AI Summary

This work addresses the tendency of large language models to rely on superficial heuristics and generate overconfident responses in medical hypothesis verification, which compromises safety and reliability. To mitigate this issue, the authors propose a two-stage verification framework that treats a model’s active abstention as a signal of uncertainty, triggering ontology-grounded selective reasoning refinement. Notably, this approach achieves knowledge graph–level performance without explicitly constructing a knowledge graph. Evaluated on MedReason and MedQA benchmarks, the method substantially improves performance, attaining a question-level accuracy of 92.5% (+9.6%) and a hypothesis-level accuracy of 96.2% (+4.2%), effectively balancing coverage and precision.

Abstract

Large language models (LLMs) often rely on shortcuts rather than systematic reasoning, raising safety concerns in medical applications. Allowing models to abstain when uncertain improves reliability but introduces a coverage accuracy tradeoff. We propose a two-stage framework for medical hypothesis verification in multiple-choice settings that manages this tradeoff through targeted ontology grounding, applied only when the model abstains. We show that abstention is not random but reflects genuine uncertainty, with abstained predictions associated with lower confidence. Across two frontier models (GPT-5.5, accessed via the Azure OpenAI API, and DeepSeek-R1), the proposed framework improves question-level accuracy by 9.6 percentage points (82.9% to 92.5%) and hypothesis-level accuracy by 4.2 percentage points (92.0% to 96.2%). Our experiments conducted on MedReason and MedQA show that abstention can be repurposed as a control signal for selective reasoning refinement, achieving knowledge-graph-level performance without explicit knowledge graph construction.

Problem

• LLM verification
• medical reasoning
• abstention
• uncertainty
• coverage-accuracy tradeoff

Innovation

• abstention
• ontology grounding
• selective refinement
• uncertainty-aware verification
• medical reasoning