LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data

2026-06-17

AI Summary

This study addresses the challenge of large language models (LLMs) struggling to reliably assess the confidence of their predictions on structured clinical data. The authors propose a calibration method that requires neither access to internal model states nor repeated inference. By quantifying the attribution disagreement between an LLM (Qwen 2.5-7B) and an XGBoost model, they introduce an Attribution Disagreement Score (ADS) as a proxy for epistemic uncertainty. This score is integrated with SHAP-based feature explanations and few-shot prompting to guide model predictions. The approach reduces ADS from 1.54 to 0.38, improves prediction accuracy from 49% to 75.3%, and lowers expected calibration error from 0.254 to 0.080, substantially enhancing the reliability and consistency of patient-level predictions and effectively mitigating the cold-start problem of LLMs on structured clinical data.

Abstract

Large language models (LLMs) are increasingly applied to structured clinical data, yet whether they can recognize the limits of their own knowledge on such tasks remains unexplored. We study this question through the lens of cross-model attribution divergence with the goal of reducing epistemic uncertainty for structured tasks, comparing Qwen 2.5 7B and XGBoost on a prediction task via attribution divergence analysis. We report four findings. First, LLM verbalized confidence is epistemically vacuous, it outputs a near-constant (0.856-0.937) regardless of whether accuracy is 49% or 75.3%, tracking prompt format rather than prediction quality. Second, the LLM exhibits an inverse difficulty effect: accuracy drops to 64.8% when XGBoost is 99% correct, but matches XGBoost (73.8% vs. 73.1%) when it is moderately uncertain. Third, few-shot examples and SHAP-derived feature evidence are orthogonal, super-additive interventions: they reduce the Attribution Disagreement Score (ADS) from 1.54 to 0.38 and improve accuracy from 49% to 75.3% without training. Fourth, a cross-model calibrator that determined LLM reliability using attribution divergence signals reduces expected calibration error from 0.254 to 0.080, replacing uninformative verbalized confidence with patient-specific reliability estimates, without accessing model internals or requiring repeated inference. We frame these findings as a cold start problem for LLMs on structured data and outline a path toward genuine epistemic self-awareness.

Problem

• epistemic blind spots
• large language models
• clinical tabular data
• attribution divergence
• self-awareness

Innovation

• attribution divergence
• epistemic uncertainty
• cross-model calibration
• structured clinical data
• LLM self-awareness