Multimodal Training to Unimodal Deployment: Leveraging Unstructured Data During Training to Optimize Structured Data Only Deployment

2026-03-23

AI Summary

This work addresses the challenge of leveraging unstructured clinical notes in electronic health records (EHR) during training to enhance the performance of models that rely solely on structured data at deployment. The authors propose a multimodal learning framework that integrates clinical notes—encoded via BioClinicalBERT—with structured features such as demographics and medical codes during training. By combining a teacher–student architecture, contrastive learning, and contrastive knowledge distillation, the approach enables the final deployed model to achieve high inference efficiency using only structured inputs, without requiring access to textual data at test time. To the best of the authors’ knowledge, this is the first method to effectively augment structured EHR representations with unstructured clinical notes while maintaining deployment constraints. Evaluated on a cohort of 3,466 children with late language emergence, the model achieves an AUROC of 0.705, significantly outperforming baseline methods (AUROC = 0.656).

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Unstructured Electronic Health Record (EHR) data, such as clinical notes, contain clinical contextual observations that are not directly reflected in structured data fields. This additional information can substantially improve model learning. However, due to their unstructured nature, these data are often unavailable or impractical to use when deploying a model. We introduce a multimodal learning framework that leverages unstructured EHR data during training while producing a model that can be deployed using only structured EHR data. Using a cohort of 3,466 children evaluated for late talking, we generated note embeddings with BioClinicalBERT and encoded structured embeddings from demographics and medical codes. A note-based teacher model and a structured-only student model were jointly trained using contrastive learning and contrastive knowledge distillation loss, producing a strong classifier (AUROC = 0.985). Our proposed model reached an AUROC of 0.705, outperforming the structured-only baseline of 0.656. These results demonstrate that incorporating unstructured data during training enhances the model's capacity to identify task-relevant information within structured EHR data, enabling a deployable structured-only phenotype model.

Problem

• multimodal training
• unimodal deployment
• structured EHR data
• unstructured EHR data
• model deployment

Innovation

• multimodal learning
• knowledge distillation
• contrastive learning
• structured EHR
• unstructured data