🤖 AI Summary
The opacity of black-box AI models in medical devices undermines clinical trust and poses regulatory compliance risks due to insufficient explainability. Method: This study pioneers a systematic integration of eXplainable Artificial Intelligence (XAI) techniques, empirical clinical human–AI interaction research, and regulatory requirements—yielding an XAI integration pathway and tiered validation framework tailored to real-world clinical workflows. Leveraging multimodal expert consensus, in situ clinical behavioral observation, AI explanation evaluation, and MHRA regulatory alignment analysis, we developed a comprehensive XAI implementation guideline spanning development, verification, and deployment phases. Contribution/Results: The guideline was formally adopted by the UK’s Medicines and Healthcare products Regulatory Agency (MHRA) as an official regulatory reference, demonstrably enhancing clinical acceptance and risk controllability. It establishes a methodological paradigm and practical standard for deploying trustworthy, clinically viable AI in healthcare.
📝 Abstract
There is a growing demand for the use of Artificial Intelligence (AI) and Machine Learning (ML) in healthcare, particularly as clinical decision support systems to assist medical professionals. However, the complexity of many of these models, often referred to as black box models, raises concerns about their safe integration into clinical settings as it is difficult to understand how they arrived at their predictions. This paper discusses insights and recommendations derived from an expert working group convened by the UK Medicine and Healthcare products Regulatory Agency (MHRA). The group consisted of healthcare professionals, regulators, and data scientists, with a primary focus on evaluating the outputs from different AI algorithms in clinical decision-making contexts. Additionally, the group evaluated findings from a pilot study investigating clinicians' behaviour and interaction with AI methods during clinical diagnosis. Incorporating AI methods is crucial for ensuring the safety and trustworthiness of medical AI devices in clinical settings. Adequate training for stakeholders is essential to address potential issues, and further insights and recommendations for safely adopting AI systems in healthcare settings are provided.