OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Prediction

2026-04-18

AI Summary

This work addresses limitations in existing ICU risk prediction methods, which often overlook the clinical semantic relationships among diagnoses during contrastive pretraining and struggle to effectively integrate multimodal information—such as clinical notes—during fine-tuning. To overcome these challenges, the authors propose OC-Distill, a two-stage framework that first leverages the hierarchical structure of ICD ontologies to construct a clinically aware contrastive learning objective, thereby modeling semantic similarity between patients. In the second stage, cross-modal knowledge distillation transfers supervisory signals from rich clinical notes into a lightweight model that relies solely on vital signs. Evaluated across multiple ICU prediction tasks on the MIMIC dataset, OC-Distill significantly outperforms state-of-the-art vital-signs-only baselines and achieves substantially improved label efficiency.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Early prediction of severe clinical deterioration and remaining length of stay can enable timely intervention and better resource allocation in high-acuity settings such as the ICU. This has driven the development of machine learning models that leverage continuous streams of vital signs and other physiological signals for real-time risk prediction. Despite their promise, existing methods have important limitations. Contrastive pretraining treats all patients as equally strong negatives, failing to capture clinically meaningful similarity between patients with related diagnoses. Meanwhile, downstream fine-tuning typically ignores complementary modalities such as clinical notes, which provide rich contextual information unavailable in physiological signals alone. To address these challenges, we propose OC-Distill, a two-stage framework that leverages multimodal supervision during training while requiring only vital signs at inference. In the first stage, we introduce an ontology-aware contrastive objective that exploits the ICD hierarchy to quantify patient similarity and learn clinically grounded representations. In the second stage, we fine-tune the pretrained encoder via cross-modal knowledge distillation, transferring complementary information from clinical notes into the model. Across multiple ICU prediction tasks on MIMIC, OC-Distill demonstrates improved label efficiency and achieves state-of-the-art performance among methods that use only vital signs at inference.

Problem

• ICU risk prediction
• contrastive learning
• cross-modal distillation
• ontology-aware
• clinical notes

Innovation

• ontology-aware contrastive learning
• cross-modal distillation
• ICU risk prediction
• knowledge distillation
• multimodal supervision