Quantifying Hallucinations in Language Language Models on Medical Textbooks

2026-02-12

arXiv.org

AI Summary

Large language models (LLMs) are prone to generating hallucinations in medical question-answering tasks, yet systematic evaluation and mitigation strategies grounded in fixed evidence sources remain lacking. This study presents the first closed-domain assessment—restricted to medical textbooks—of hallucination rates in open-source LLMs using zero-shot prompting, expert clinician evaluations, and statistical analyses (Cohen’s κ and Kendall’s τ) to quantify hallucinations and examine their relationship with clinical utility. Results reveal a hallucination rate of 19.7% for LLaMA-70B-Instruct, with lower hallucination rates strongly associated with higher clinical usefulness scores. High inter-rater agreement among physicians underscores the current unsuitability of these models for unsupervised clinical deployment.

Abstract

Hallucinations, the tendency for large language models to provide responses with factually incorrect and unsupported claims, is a serious problem within natural language processing for which we do not yet have an effective solution to mitigate against. Existing benchmarks for medical QA rarely evaluate this behavior against a fixed evidence source. We ask how often hallucinations occur on textbook-grounded QA and how responses to medical QA prompts vary across models. We conduct two experiments: the first experiment to determine the prevalence of hallucinations for a prominent open source large language model (LLaMA-70B-Instruct) in medical QA given novel prompts, and the second experiment to determine the prevalence of hallucinations and clinician preference to model responses. We observed, in experiment one, with the passages provided, LLaMA-70B-Instruct hallucinated in 19.7\% of answers (95\% CI 18.6 to 20.7) even though 98.8\% of prompt responses received maximal plausibility, and observed in experiment two, across models, lower hallucination rates aligned with higher usefulness scores ($ρ=-0.71$, $p=0.058$). Clinicians produced high agreement (quadratic weighted $κ=0.92$) and ($τ_b=0.06$ to $0.18$, $κ=0.57$ to $0.61$) for experiments 1 and ,2 respectively

Problem

• hallucination
• large language models
• medical QA
• factuality
• textbook-grounded

Innovation

• hallucination quantification
• medical textbook grounding
• clinician evaluation
• large language models
• zero-shot QA