Safety challenges of AI in medicine in the era of large language models

2024-09-11

AI Summary

This study systematically uncovers novel safety risks of large language models (LLMs) in healthcare applications, spanning functional failure, clinician–patient communication distortions, training data contamination, and erosion of clinical trust. Methodologically, it innovatively deconstructs LLM-specific risks through dual lenses—functional reliability and human–AI communication efficacy—integrating general AI safety concerns with systemic healthcare impacts. Drawing on a systematic literature review and causal risk attribution analysis, the work synthesizes insights from medical human factors engineering, explainable AI, clinical workflow modeling, and trust theory to identify 12 emergent risk categories and elucidate how LLMs exacerbate preexisting AI vulnerabilities. The study proposes a multi-layered, “trustworthy integration”–oriented safety framework, offering theoretical foundations and actionable pathways for regulatory guideline development, standardized safety evaluation protocols, and safe clinical deployment practices.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Recent advancements in artificial intelligence (AI), particularly in large language models (LLMs), have unlocked significant potential to enhance the quality and efficiency of medical care. By introducing a novel way to interact with AI and data through natural language, LLMs offer new opportunities for medical practitioners, patients, and researchers. However, as AI and LLMs become more powerful and especially achieve superhuman performance in some medical tasks, public concerns over their safety have intensified. These concerns about AI safety have emerged as the most significant obstacles to the adoption of AI in medicine. In response, this review examines emerging risks in AI utilization during the LLM era. First, we explore LLM-specific safety challenges from functional and communication perspectives, addressing issues across data collection, model training, and real-world application. We then consider inherent safety problems shared by all AI systems, along with additional complications introduced by LLMs. Last, we discussed how safety issues of using AI in clinical practice and healthcare system operation would undermine trust among patient, clinicians and the public, and how to build confidence in these systems. By emphasizing the development of safe AI, we believe these technologies can be more rapidly and reliably integrated into everyday medical practice to benefit both patients and clinicians.

Problem

• AI safety
• large language models
• medical field

Innovation

• Large Language Models
• Healthcare AI Safety
• Patient Interaction Risks