Multimodal Deep Learning for Prediction of Progression-Free Survival in Patients with Neuroendocrine Tumors Undergoing 177Lu-based Peptide Receptor Radionuclide Therapy

2025-11-07

AI Summary

This study addresses the challenge of predicting progression-free survival (PFS) in patients with neuroendocrine tumors (NETs) following ¹⁷⁷Lu-DOTATOC peptide receptor radionuclide therapy (PRRT), to support personalized treatment decisions. We propose a novel multimodal deep learning framework that integrates pre-trained 3D CT imaging, somatostatin receptor (SSTR)-PET quantitative features, and clinical laboratory biomarkers. Methodologically, we introduce CT branch pre-training and a cross-modal adaptive feature fusion mechanism, augmented by gradient-weighted class activation mapping (Grad-CAM) and feature importance analysis to enhance model interpretability. Evaluated on an independent validation cohort, our model achieves an AUROC of 0.72 ± 0.01 and an AUPRC of 0.80 ± 0.01—significantly outperforming unimodal baselines (p < 0.01). The framework enables clinically translatable, risk-stratified PFS prediction with robust generalizability and transparency.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Peptide receptor radionuclide therapy (PRRT) is an established treatment for metastatic neuroendocrine tumors (NETs), yet long-term disease control occurs only in a subset of patients. Predicting progression-free survival (PFS) could support individualized treatment planning. This study evaluates laboratory, imaging, and multimodal deep learning models for PFS prediction in PRRT-treated patients. In this retrospective, single-center study 116 patients with metastatic NETs undergoing 177Lu-DOTATOC were included. Clinical characteristics, laboratory values, and pretherapeutic somatostatin receptor positron emission tomography/computed tomographies (SR-PET/CT) were collected. Seven models were trained to classify low- vs. high-PFS groups, including unimodal (laboratory, SR-PET, or CT) and multimodal fusion approaches. Explainability was evaluated by feature importance analysis and gradient maps. Forty-two patients (36%) had short PFS (<1 year), 74 patients long PFS (>1 year). Groups were similar in most characteristics, except for higher baseline chromogranin A (p = 0.003), elevated gamma-GT (p = 0.002), and fewer PRRT cycles (p<0.001) in short-PFS patients. The Random Forest model trained only on laboratory biomarkers reached an AUROC of 0.59 +- 0.02. Unimodal three-dimensional convolutional neural networks using SR-PET or CT performed worse (AUROC 0.42 +- 0.03 and 0.54 +- 0.01, respectively). A multimodal fusion model laboratory values, SR-PET, and CT -augmented with a pretrained CT branch - achieved the best results (AUROC 0.72 +- 0.01, AUPRC 0.80 +- 0.01). Multimodal deep learning combining SR-PET, CT, and laboratory biomarkers outperformed unimodal approaches for PFS prediction after PRRT. Upon external validation, such models may support risk-adapted follow-up strategies.

Problem

• Predicting progression-free survival for neuroendocrine tumor patients undergoing PRRT
• Evaluating multimodal deep learning models combining imaging and laboratory data
• Developing individualized treatment planning tools through survival prediction models

Innovation

• Multimodal deep learning combines SR-PET, CT, and laboratory biomarkers
• Fusion model uses pretrained CT branch to enhance performance
• Deep learning predicts progression-free survival for neuroendocrine tumors