Foundation Models for Clinical Records at Health System Scale

2025-07-01

AI Summary

Generative pretraining for electronic health record (EHR) sequence modeling remains underexplored. Method: We propose an autoregressive generative pretraining framework tailored to large-scale EHRs, predicting the complete clinical event sequence of a patient’s next visit—enabling joint modeling of heterogeneous medical data and zero-shot long-term disease risk prediction. Innovatively, we adopt a “next-visit event prediction” objective and introduce a repetition-aware regularization mechanism to mitigate inflated evaluation metrics caused by event redundancy in EHR foundation models. Leveraging tokenized clinical event sequences and a large-scale autoregressive architecture, the model performs zero-shot inference without fine-tuning. Results: On 2-year and 5-year dementia and knee osteoarthritis incidence prediction tasks, it matches the performance of fully fine-tuned masked language modeling Transformers, demonstrating superior capacity for capturing complex longitudinal clinical dependencies.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Large-scale pretraining has transformed modeling of language and other data types, but its potential remains underexplored in healthcare with structured electronic health records (EHRs). We present a novel generative pretraining strategy for sequential EHR data using next-visit event prediction. Our model learns to autoregressively generate various tokenized clinical events for the next visit based on patient history and inherently handles the joint prediction of heterogeneous data types. Additionally, we introduce regularization on predicting repeated events and highlight a key pitfall in EHR-based foundation model evaluations: repeated event tokens can inflate performance metrics when new onsets are not distinguished from subsequent occurrences. Our model is evaluated via zero-shot prediction for forecasting dementia and knee osteoarthritis incidence within 2 and 5 years, and the model performance rivals a fully fine-tuned masked pretrained Transformer baseline, demonstrating that our approach captures complex clinical dependencies without requiring costly task-specific fine-tuning.

Problem

• Develops generative pretraining for sequential EHR data
• Addresses performance inflation from repeated event tokens
• Evaluates zero-shot prediction for dementia and osteoarthritis

Innovation

• Generative pretraining for sequential EHR data
• Regularization on predicting repeated events
• Zero-shot prediction for clinical outcomes