The strength of clinical evidence is recoverable from language model representations but not from their stated grades

2026-06-27

AI Summary

This study addresses the challenge that clinical large language models (LLMs) often fail to accurately convey the strength of medical evidence in their summaries, undermining their reliability. The authors construct a dataset of 45,134 clinical claims and systematically evaluate 22 open-source LLMs by recovering evidence levels from internal model representations using linear probes and comparing them against the models’ explicit evidence ratings. They find, for the first time, that LLMs implicitly encode recoverable signals of evidence strength (median AUROC 71.8), which are independent of factual correctness and primarily driven by lexical features. Although this signal lacks generalizability across topics or modeling frameworks, it effectively identifies weakly supported claims (AUROC 69.2). In contrast, the models’ explicitly stated evidence levels perform near chance, offering little practical utility.

Abstract

Large language models (LLMs) increasingly summarize clinical evidence, where a claim's weight depends on how strongly it is supported. Yet these models convey confidence poorly, and properties they never state, such as truth, are often readable from their activations. Whether a clinical model registers evidence strength, distinct from truth, and states it when asked is untested, and any such signal could be lexical. We compiled 45,134 clinical claims from six public sources, harmonized 20,611 into a four-level evidence grade under three independent frameworks, and tested 22 local, open-weight LLMs from several developers (0.6-70 billion parameters; general, medical, and reasoning), with lexical, truth, and cross-framework controls. A linear estimator recovered the grade in every model (median AUROC 71.8), yet decodability did not rise with scale and was weakest in reasoning models. The grade the models stated fell to chance, 25-27 percentage points below the estimator. The recoverable signal was largely lexical and did not transfer across topics or frameworks, yet it was distinct from factual truth and still flagged weakly supported claims (AUROC 69.2). Clinical LLMs thus carry an ordered evidence-strength signal they do not express, so their stated grades fail to convey a claim's support even when it is recoverable from their representations and text.

Problem

• clinical evidence
• evidence strength
• large language models
• model confidence
• representation decoding

Innovation

• evidence strength
• large language models
• linear probing
• clinical claims
• model interpretability