Uncertainty Is Not a Safety Net for Clinical VQA, but Can It Anticipate Model Failure?

2026-06-15

AI Summary

This study addresses the critical challenge in clinical visual question answering (VQA) that existing uncertainty estimation methods often fail to provide reliable warnings precisely when model performance is weakest, thereby hindering safe deployment. The authors systematically evaluate eight uncertainty quantification approaches across twelve vision-language models and introduce a NOTA perturbation stress test to assess how well uncertainty signals predict model failure. Their findings reveal that uncertainty quality is strongly correlated with model accuracy rather than serving as an independent reliability indicator; however, uncertainty derived from original inputs effectively predicts model vulnerability under perturbations. While current methods are insufficient as standalone safety nets, they show promise as diagnostic tools for flagging high-risk predictions, supporting the adoption of perturbation-based evaluation paradigms in clinical settings.

Abstract

Safe deployment of clinical vision-language models (VLMs) requires reliable uncertainty estimation (UE): a signal indicating when predictions should be trusted or escalated to a clinician. We test whether current UE methods actually deliver this signal. Benchmarking 8 methods across 12 VLMs on clinical visual question-answering (VQA), we find that UE quality is not an intrinsic property of the UE method: it tracks model accuracy, degrading precisely where the model performance is weakest, and therefore where reliability is most needed. When we stress-test models by hiding the correct option among the multiple-choice answers (NOTA perturbations), accuracy collapses while uncertainty barely changes, leaving models systematically miscalibrated. Yet, we find that uncertainty on the unperturbed input reliably anticipates which predictions will collapse under NOTA, indicating that UE in current VLMs carries diagnostic information about model fragility. Our results position UE as a diagnostic tool for identifying fragile predictions and motivate perturbation-based evaluation as a path toward safe clinical deployment.

Problem

• uncertainty estimation
• clinical VQA
• model failure
• vision-language models
• reliability

Innovation

• uncertainty estimation
• clinical VQA
• vision-language models
• model calibration
• perturbation-based evaluation