The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing

2026-06-23

AI Summary

Current autonomous AI prescribing systems suffer from low clinical adoption and ambiguous accountability due to inadequate representation of uncertainty, uncalibrated confidence thresholds, and insufficient reasoning transparency. This work proposes a novel architecture that reformulates autonomy as high-supervision decision support by explicitly incorporating physicians’ preferences for handling uncertainty into system design. The framework integrates calibrated confidence modeling, decomposition of aleatoric and epistemic uncertainties, generation of interpretable reasoning pathways, and optimized human–AI interaction. Empirical evaluation with 136 U.S.-based prescribing physicians demonstrates that this approach significantly enhances clinical acceptance: physicians are willing to assume additional responsibility only when the system provides both a confidence escalation mechanism and transparent reasoning, and they adapt their decision-making strategies according to the type of uncertainty presented.

Abstract

Autonomous AI systems are transitioning from advisory to autonomous roles for medication prescriptions. Recent United States bill H.R. 238 and Utah's prescription-renewal pilot both authorize AI to prescribe medications in an agentic capacity. While some regulatory guidelines suggest aggregate model performance metrics for clearance, they do not require i) calibrated per-prediction confidence for action-gated thresholds, ii) differentiated communication of uncertainty arising from model ignorance (epistemic) versus genuine clinical ambiguity (aleatoric), and iii) inferential transparency at the moment of decision that allows for liability allocation. Here, we present a regulatory and technical argument (tested with a survey of 136 U.S. prescribing clinicians) positioning these as minimum architectural requirements for safe autonomous prescribing. Our results suggest prescribing clinicians i) would not permit autonomous prescribing without a calibrated confidence-based escalation mechanism, ii) preferred a competing-options summary when uncertainty was aleatoric but shifted to abstention when uncertainty was epistemic, and iii) were only willing to accept additional liability when inferential transparency enabled a substantive judgment under acknowledged uncertainty. These findings indicate our recommended architectural features would encourage higher rates of clinician adoption, largely through collapsing much of what "autonomy" conventionally means. A system meeting these requirements would function less as an autonomous agent and more as a heavily supervised decision-support tool. As legislation and state pilots proceed, our technical argument backed by clinician perspectives provides opportunities for regulation to constrain the degree of autonomy ethically granted to AI in prescribing while aligning liability with the institutional actors who control system design and deployment.

Problem

• autonomous AI prescribing
• clinical uncertainty
• liability allocation
• trust in AI
• regulatory requirements

Innovation

• calibrated confidence
• epistemic vs aleatoric uncertainty
• inferential transparency
• autonomous prescribing
• liability allocation