Multi-modal AI for comprehensive breast cancer prognostication

2024-10-28

arXiv.org

AI Summary

Current breast cancer prognostic tools—such as Oncotype DX—exhibit suboptimal accuracy, particularly for triple-negative breast cancer (TNBC), where no clinically validated predictive biomarkers exist. To address this gap, we propose a multimodal AI model integrating digitized H&E-stained histopathological images with clinical covariates. Our approach leverages a pan-cancer vision transformer (ViT) pretrained via self-supervised learning, followed by multimodal feature fusion and Cox proportional hazards modeling for survival prediction. The model was rigorously validated across five independent cohorts (n = 3,502), achieving an overall concordance index (C-index) of 0.71—significantly outperforming Oncotype DX (0.67 vs. 0.61). Notably, it attains a C-index of 0.71 specifically in TNBC, representing the first AI-based prognostic tool for TNBC with demonstrated clinical utility.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Treatment selection in breast cancer is guided by molecular subtypes and clinical characteristics. However, current tools including genomic assays lack the accuracy required for optimal clinical decision-making. We developed a novel artificial intelligence (AI)-based approach that integrates digital pathology images with clinical data, providing a more robust and effective method for predicting the risk of cancer recurrence in breast cancer patients. Specifically, we utilized a vision transformer pan-cancer foundation model trained with self-supervised learning to extract features from digitized H&E-stained slides. These features were integrated with clinical data to form a multi-modal AI test predicting cancer recurrence and death. The test was developed and evaluated using data from a total of 8,161 female breast cancer patients across 15 cohorts originating from seven countries. Of these, 3,502 patients from five cohorts were used exclusively for evaluation, while the remaining patients were used for training. Our test accurately predicted our primary endpoint, disease-free interval, in the five evaluation cohorts (C-index: 0.71 [0.68-0.75], HR: 3.63 [3.02-4.37, p<0.001]). In a direct comparison (n=858), the AI test was more accurate than Oncotype DX, the standard-of-care 21-gene assay, achieving a C-index of 0.67 [0.61-0.74] versus 0.61 [0.49-0.73], respectively. Additionally, the AI test added independent prognostic information to Oncotype DX in a multivariate analysis (HR: 3.11 [1.91-5.09, p<0.001)]). The test demonstrated robust accuracy across major molecular breast cancer subtypes, including TNBC (C-index: 0.71 [0.62-0.81], HR: 3.81 [2.35-6.17, p=0.02]), where no diagnostic tools are currently recommended by clinical guidelines. These results suggest that our AI test improves upon the accuracy of existing prognostic tests, while being applicable to a wider range of patients.

Problem

• Improves breast cancer recurrence prediction accuracy.
• Integrates pathology images with clinical data.
• Outperforms standard genomic assays like Oncotype DX.

Innovation

• Integrates digital pathology with clinical data
• Uses vision transformer for feature extraction
• Outperforms Oncotype DX in accuracy