One Loss to Rule Them All: Marked Time-to-Event for Structured EHR Foundation Models

2026-01-31

AI Summary

This work proposes ORA, a novel foundation model for electronic health records (EHR) that introduces the time-to-event pretraining objective to jointly model the timing of clinical events and their associated continuous measurements. In contrast to existing EHR foundation models that primarily rely on next-item prediction and struggle to capture temporal dynamics and continuous-valued observations, ORA overcomes the limitation of predicting only event types by explicitly incorporating event timestamps and numerical values into a unified framework. The approach is architecture-agnostic and seamlessly integrates discrete clinical events with continuous physiological measurements. Extensive experiments across multiple datasets, downstream tasks, and backbone architectures demonstrate that ORA consistently achieves superior performance in classification, regression, and temporal prediction tasks, highlighting its enhanced generalization capability.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Other Foundations of Cognitive Modeling & Systems
• Knowledge Representation and Reasoning: Other Foundations of Knowledge Representation & Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Clinical events captured in Electronic Health Records (EHR) are irregularly sampled and may consist of a mixture of discrete events and numerical measurements, such as laboratory values or treatment dosages. The sequential nature of EHR, analogous to natural language, has motivated the use of next-token prediction to train prior EHR Foundation Models (FMs) over events. However, this training fails to capture the full structure of EHR. We propose ORA, a marked time-to-event pretraining objective that jointly models event timing and associated measurements. Across multiple datasets, downstream tasks, and model architectures, this objective consistently yields more generalizable representations than next-token prediction and pretraining losses that ignore continuous measurements. Importantly, the proposed objective yields improvements beyond traditional classification evaluation, including better regression and time-to-event prediction. Beyond introducing a new family of FMs, our results suggest a broader takeaway: pretraining objectives that account for EHR structure are critical for expanding downstream capabilities and generalizability

Problem

• Electronic Health Records
• Foundation Models
• Time-to-Event Prediction
• Pretraining Objective
• Structured Clinical Data

Innovation

• marked time-to-event
• EHR foundation models
• pretraining objective
• structured clinical data
• joint modeling