Multimodal Deep Learning for Stroke Prediction and Detection using Retinal Imaging and Clinical Data

2025-05-05

AI Summary

Stroke early prediction and acute-phase detection remain costly and heavily reliant on expensive neuroimaging. Method: We propose a multimodal deep learning framework integrating retinal optical coherence tomography (OCT), infrared reflectance images, and structured clinical data (demographics, vital signs, diagnosis codes). To address limited labeled retinal image data, we introduce self-supervised pretraining for retinal imaging modeling—novel in this domain—and design an end-to-end multimodal fusion architecture to overcome representational limitations of unimodal imaging. Contribution/Results: This work provides the first systematic validation of retinal imaging as an independent predictor of long-term stroke risk. Experiments demonstrate that our model achieves an AUROC improvement of 5% over unimodal imaging baselines and 8% over state-of-the-art models. These results confirm that retinal microvascular alterations serve as a valid, noninvasive, and cost-effective biomarker for cerebrovascular pathology, establishing a new paradigm for scalable stroke risk stratification and acute identification.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Stroke is a major public health problem, affecting millions worldwide. Deep learning has recently demonstrated promise for enhancing the diagnosis and risk prediction of stroke. However, existing methods rely on costly medical imaging modalities, such as computed tomography. Recent studies suggest that retinal imaging could offer a cost-effective alternative for cerebrovascular health assessment due to the shared clinical pathways between the retina and the brain. Hence, this study explores the impact of leveraging retinal images and clinical data for stroke detection and risk prediction. We propose a multimodal deep neural network that processes Optical Coherence Tomography (OCT) and infrared reflectance retinal scans, combined with clinical data, such as demographics, vital signs, and diagnosis codes. We pretrained our model using a self-supervised learning framework using a real-world dataset consisting of $37$ k scans, and then fine-tuned and evaluated the model using a smaller labeled subset. Our empirical findings establish the predictive ability of the considered modalities in detecting lasting effects in the retina associated with acute stroke and forecasting future risk within a specific time horizon. The experimental results demonstrate the effectiveness of our proposed framework by achieving $5$% AUROC improvement as compared to the unimodal image-only baseline, and $8$% improvement compared to an existing state-of-the-art foundation model. In conclusion, our study highlights the potential of retinal imaging in identifying high-risk patients and improving long-term outcomes.

Problem

• Developing multimodal deep learning for stroke prediction using retinal imaging and clinical data
• Exploring cost-effective retinal imaging as an alternative to expensive stroke diagnostic methods
• Improving stroke risk prediction accuracy by combining OCT scans with patient clinical data

Innovation

• Multimodal deep learning for stroke prediction
• Combines retinal imaging with clinical data
• Self-supervised pretraining on large dataset