Hierarchical Modeling of ICD Codes in EHR Foundation Models

2026-06-13

AI Summary

This work addresses the limitation of existing electronic health record (EHR) foundation models that treat ICD diagnosis codes as flat tokens, thereby neglecting their intrinsic clinical hierarchical structure and underutilizing semantic information. To overcome this, the study introduces the ICD-10-CM hierarchy as an inductive bias into EHR modeling for the first time, proposing a hierarchy-enhanced Transformer and a hierarchy-aware graph neural network that jointly leverage diagnosis co-occurrence patterns and multi-granular ICD codes. The model is pretrained on MIMIC-IV and evaluated via frozen probing on eICU for cross-dataset generalization. Results demonstrate consistent and significant improvements over flat-code baselines in both in-domain and cross-domain settings, enhancing downstream prediction performance and yielding more semantically coherent embedding spaces, thus validating the broad utility of hierarchical modeling across diverse tasks and architectures.

Abstract

Electronic health record foundation models typically treat ICD diagnosis codes as flat tokens, overlooking the clinically meaningful hierarchical structure that captures disease families, subcategories, and fine-grained diagnostic detail. As a result, existing EHR representation learning methods do not explicitly exploit the hierarchical structure already present in the coding system. In this work, we study ICD-10-CM hierarchy as a general inductive bias for clinical representation learning. We investigate two complementary mechanisms for incorporating hierarchy: first, by augmenting diagnosis sequences in a BERT-style transformer with tokens corresponding to different levels of the ICD hierarchy, and second, by injecting hierarchy into graph-based code representations through hierarchy-aware edges combined with diagnosis co-occurrence structure. Across these settings, we evaluate whether explicit hierarchy improves downstream prediction, which levels of the hierarchy are most useful, whether hierarchy encoding improves transfer across datasets, and how hierarchy reshapes embedding similarity structure. We conduct experiments on two large-scale real-world clinical datasets: MIMIC-IV, used for pretraining and in-domain evaluation, and eICU, used to assess cross-dataset transfer via frozen encoder probing. Our findings show that explicitly encoding ICD hierarchy improves over flat code representations in both in-domain and cross-dataset settings, while revealing that the most useful level of hierarchy depends on both the task and the modeling approach. More broadly, we focus on hierarchy-aware EHR representation learning and show that the benefits of encoding hierarchy are generalizable across modeling settings and hierarchy levels.

Problem

• ICD codes
• hierarchical structure
• EHR foundation models
• representation learning
• clinical data

Innovation

• hierarchical modeling
• ICD codes
• EHR foundation models
• representation learning
• inductive bias