Multi-stage intermediate fusion for multimodal learning to classify non-small cell lung cancer subtypes from CT and PET

2025-01-21

AI Summary

To address the clinical need for non-invasive, accurate subtyping of non-small cell lung cancer (NSCLC), this study proposes a multi-stage intermediate fusion framework that jointly leverages CT and PET imaging. Voxel-wise cross-modal fusion is performed at multiple hierarchical levels of multi-scale feature extraction, preserving both modality complementarity and spatial consistency. This work presents the first systematic validation demonstrating that intermediate fusion significantly outperforms early/late fusion strategies—as well as the sole existing intermediate fusion method—in NSCLC subtyping. An ablation-driven optimization of fusion stages and end-to-end joint training are employed. Experimental results show an accuracy of 0.724 and an AUC of 0.681, substantially surpassing unimodal baselines and all competing fusion approaches. The proposed framework establishes a robust, interpretable, non-invasive diagnostic paradigm for NSCLC subtyping.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Accurate classification of histological subtypes of non-small cell lung cancer (NSCLC) is essential in the era of precision medicine, yet current invasive techniques are not always feasible and may lead to clinical complications. This study presents a multi-stage intermediate fusion approach to classify NSCLC subtypes from CT and PET images. Our method integrates the two modalities at different stages of feature extraction, using voxel-wise fusion to exploit complementary information across varying abstraction levels while preserving spatial correlations. We compare our method against unimodal approaches using only CT or PET images to demonstrate the benefits of modality fusion, and further benchmark it against early and late fusion techniques to highlight the advantages of intermediate fusion during feature extraction. Additionally, we compare our model with the only existing intermediate fusion method for histological subtype classification using PET/CT images. Our results demonstrate that the proposed method outperforms all alternatives across key metrics, with an accuracy and AUC equal to 0.724 and 0.681, respectively. This non-invasive approach has the potential to significantly improve diagnostic accuracy, facilitate more informed treatment decisions, and advance personalized care in lung cancer management.

Problem

• Non-Small Cell Lung Cancer
• Non-Invasive Diagnosis
• Differential Typing

Innovation

• Multi-step Method
• CT-PET Image Fusion
• Non-Small Cell Lung Cancer Identification