Problem
Research questions and friction points this paper is trying to address.
clinical question answering
verbatim citation
verifiable by construction
Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question Answering
🤖 AI Summary
研究通过构建细粒度引用验证临床问答系统，评估了12种大语言模型在提供和验证引用方面的能力，揭示了当前模型存在的能力差距。
📝 Abstract
Large language models (LLMs) have been widely adopted for clinical question answering (QA). Current systems can attach citations to their answers, but these often point to broad texts, leaving time-pressed clinicians unable to verify them efficiently. An alternative is to ensure that responses are verifiable by construction: providing fine-grained verbatim quotes from reference material that substantiate claims, so users can verify an answer without opening other documents. In this paper, we evaluate the ability of current models to perform this task end-to-end: from providing citations for every factual claim, to producing verbatim quotes, to ensuring that those quotes fully substantiate the claims. To do so, we build a standardized harness over four clinical practice guidelines and evaluate twelve LLMs on 222 synthetic clinical questions, measuring each of these stages separately. We find that most models can attach verbatim quotes to over 90% of their claims from prompting alone, apart from some lightweight models such as claude-haiku-4.5. Yet these quotes often fail to substantiate every detail of the claims they accompany. For instance, claude-opus-5 produces verbatim quotes for 98.0% of its claims, but fully substantiates only 37.1%. Our work provides insights into the current capability gap of LLMs in building verifiable clinical QA systems, along with artifacts for future research.
Innovation
Methods, ideas, or system contributions that make the work stand out.
claim-level evaluation
verbatim citation
clinical question answering
🔎 Similar Papers
No similar papers found.
