Exploring Scaling Laws for EHR Foundation Models

2025-05-29

AI Summary

Prior work lacks systematic understanding of scaling laws for electronic health record (EHR) foundation models, despite their critical role in clinical AI. EHR data exhibit strong temporal dependencies and structural heterogeneity, posing unique challenges for scalable modeling. Method: We construct multi-scale Transformer-based pretraining models on MIMIC-IV, systematically varying compute (FLOPs), parameter count, and dataset size, and quantitatively evaluate their impact on clinical prediction performance. Contribution/Results: We discover that EHR foundation models follow LLM-like scaling behavior: performance peaks quadratically under fixed FLOPs, while clinical utility scales as a power law with model size. This work establishes the first predictive, resource-efficient scaling paradigm for EHR models—improving downstream task performance by up to 12.7%—and provides both theoretical foundations and empirical guidelines for designing, training, and deploying medical foundation models.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Humans and AI: Other Foundations of Human Computation & AI
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

The emergence of scaling laws has profoundly shaped the development of large language models (LLMs), enabling predictable performance gains through systematic increases in model size, dataset volume, and compute. Yet, these principles remain largely unexplored in the context of electronic health records (EHRs) -- a rich, sequential, and globally abundant data source that differs structurally from natural language. In this work, we present the first empirical investigation of scaling laws for EHR foundation models. By training transformer architectures on patient timeline data from the MIMIC-IV database across varying model sizes and compute budgets, we identify consistent scaling patterns, including parabolic IsoFLOPs curves and power-law relationships between compute, model parameters, data size, and clinical utility. These findings demonstrate that EHR models exhibit scaling behavior analogous to LLMs, offering predictive insights into resource-efficient training strategies. Our results lay the groundwork for developing powerful EHR foundation models capable of transforming clinical prediction tasks and advancing personalized healthcare.

Problem

• Investigates scaling laws for EHR foundation models
• Explores performance gains via model size and data volume
• Identifies scaling patterns in EHR models akin to LLMs

Innovation

• Transformer architectures on EHR data
• Scaling laws for EHR foundation models
• Resource-efficient EHR training strategies