Generative Medical Event Models Improve with Scale

2025-08-16

AI Summary

This study addresses the challenge of modeling large-scale longitudinal clinical event data by proposing CoMET, a generative foundation model. Trained on the Epic Cosmos dataset, CoMET employs a decoder-only Transformer architecture and performs autoregressive pretraining on 115 billion discrete clinical events—first empirically validating and leveraging power-law scaling laws for computationally optimal model scaling in healthcare sequences. Its key contribution is zero-shot generalization across 78 diverse clinical tasks—including diagnosis prediction, disease prognosis, and healthcare operations—without task-specific fine-tuning, matching or surpassing supervised task-specific models. Performance improves consistently and stably with increasing parameter count across all metrics. Results demonstrate that generative modeling effectively captures the complex temporal dynamics of patient clinical trajectories, establishing a scalable, general-purpose foundation model paradigm for personalized medicine.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Other Foundations of Cognitive Modeling & Systems

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Realizing personalized medicine at scale calls for methods that distill insights from longitudinal patient journeys, which can be viewed as a sequence of medical events. Foundation models pretrained on large-scale medical event data represent a promising direction for scaling real-world evidence generation and generalizing to diverse downstream tasks. Using Epic Cosmos, a dataset with medical events from de-identified longitudinal health records for 16.3 billion encounters over 300 million unique patient records from 310 health systems, we introduce the Cosmos Medical Event Transformer ( CoMET) models, a family of decoder-only transformer models pretrained on 118 million patients representing 115 billion discrete medical events (151 billion tokens). We present the largest scaling-law study for medical event data, establishing a methodology for pretraining and revealing power-law scaling relationships for compute, tokens, and model size. Based on this, we pretrained a series of compute-optimal models with up to 1 billion parameters. Conditioned on a patient's real-world history, CoMET autoregressively generates the next medical event, simulating patient health timelines. We studied 78 real-world tasks, including diagnosis prediction, disease prognosis, and healthcare operations. Remarkably for a foundation model with generic pretraining and simulation-based inference, CoMET generally outperformed or matched task-specific supervised models on these tasks, without requiring task-specific fine-tuning or few-shot examples. CoMET's predictive power consistently improves as the model and pretraining scale. Our results show that CoMET, a generative medical event foundation model, can effectively capture complex clinical dynamics, providing an extensible and generalizable framework to support clinical decision-making, streamline healthcare operations, and improve patient outcomes.

Problem

• Develop generative models for simulating patient medical event sequences
• Scale pretraining for medical event data to improve predictive accuracy
• Generalize foundation models to diverse clinical tasks without fine-tuning

Innovation

• Decoder-only transformer models pretrained on medical events
• Largest scaling-law study for medical event data
• Generative model for simulating patient health timelines