A Comprehensive Survey of Foundation Models in Medicine

2024-06-15

arXiv.org

AI Summary

The field of foundation models (FMs) in medicine lacks a systematic, domain-wide survey. Method: This work introduces the first unified taxonomy and knowledge graph covering clinical NLP, medical imaging, graph learning, and multi-omics. Leveraging bibliometric analysis, technical lineage modeling, and cross-modal capability evaluation, it comprehensively analyzes learning paradigms, evolutionary trajectories, and application bottlenecks of representative architectures—including BERT, GPT, Med-PaLM, and RadFM. Contribution/Results: We propose a novel paradigm framework for clinical large language models and multimodal medical AI; identify seven critical application gaps and five key directions for trustworthy AI research; and uncover deployment barriers and ethical risks. This study delivers the first structured roadmap to guide both theoretical advancement and clinical translation of medical FMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Cognitive Modeling & Cognitive Systems: Other Foundations of Cognitive Modeling & Systems

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Foundation models (FMs) are large-scale deep learning models that are developed using large datasets and self-supervised learning methods. These models serve as a base for different downstream tasks, including healthcare. FMs have been adopted with great success across various domains within healthcare. Existing healthcare-based surveys have not yet included all of these domains. Therefore, we provide a detailed survey of FMs in healthcare. We focus on the history, learning strategies, flagship models, applications, and challenges of FMs. We explore how FMs such as the BERT and GPT families are reshaping various healthcare domains, including clinical large language models, medical image analysis, and omics. Furthermore, we provide a detailed taxonomy of healthcare applications facilitated by FMs, such as clinical NLP, medical computer vision, graph learning, and other biology-related tasks. Despite the promising opportunities FMs provide, they also have several associated challenges, which are explained in detail. We also outline open research issues and potential lessons learned to provide researchers and practitioners with insights into the capabilities of FMs in healthcare to advance their deployment and mitigate associated risks.

Problem

• Foundation Models in Healthcare
• Comprehensive Review
• Challenges and Applications

Innovation

• Foundation Models in Healthcare
• Medical Language Processing
• Imaging and Genomics