Multi-Modal AI for Remote Patient Monitoring in Cancer Care

2025-11-30

AI Summary

To address the monitoring gap for adverse events and passive home-based management during outpatient intervals in systemic cancer therapy, this study develops the first multimodal AI remote monitoring framework tailored to real-world irregular time-series data. The framework integrates asynchronous, sparse, and incomplete data—including wearable-derived vital signs (e.g., heart rate), structured patient-reported questionnaires, and clinical event logs—and introduces a novel multimodal fusion model explicitly designed for non-uniformly sampled temporal sequences. Key predictive features identified include prior treatment history, diurnal maximum heart rate, and self-assessed health items. The model generates dynamic risk escalation curves to enable early clinical warning. In a prospective cohort of 84 patients comprising over 2.1 million data points, the model achieves an AUROC of 0.70 and 83.9% risk prediction accuracy, demonstrating its clinical feasibility and potential for proactive intervention.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

For patients undergoing systemic cancer therapy, the time between clinic visits is full of uncertainties and risks of unmonitored side effects. To bridge this gap in care, we developed and prospectively trialed a multi-modal AI framework for remote patient monitoring (RPM). This system integrates multi-modal data from the HALO-X platform, such as demographics, wearable sensors, daily surveys, and clinical events. Our observational trial is one of the largest of its kind and has collected over 2.1 million data points (6,080 patient-days) of monitoring from 84 patients. We developed and adapted a multi-modal AI model to handle the asynchronous and incomplete nature of real-world RPM data, forecasting a continuous risk of future adverse events. The model achieved an accuracy of 83.9% (AUROC=0.70). Notably, the model identified previous treatments, wellness check-ins, and daily maximum heart rate as key predictive features. A case study demonstrated the model's ability to provide early warnings by outputting escalating risk profiles prior to the event. This work establishes the feasibility of multi-modal AI RPM for cancer care and offers a path toward more proactive patient support.(Accepted at Europe NeurIPS 2025 Multimodal Representation Learning for Healthcare Workshop)

Problem

• Developed multi-modal AI for remote cancer patient monitoring
• Forecasted adverse event risk using real-world asynchronous data
• Achieved 83.9% accuracy in predicting future adverse events

Innovation

• Multi-modal AI integrates diverse patient data sources
• AI forecasts continuous risk of adverse events
• Model provides early warnings through escalating risk profiles