🤖 AI Summary
This study addresses the challenge of effectively fusing medical images with clinical and demographic tabular data by proposing a hybrid multimodal fusion approach based on the Mamba architecture. The method employs a Vision Mamba to extract lesion image features and a Tabular Mamba to integrate image-derived prediction probabilities with structured clinical data, enabling end-to-end training for cancer classification. Notably, this work is the first to introduce Mamba into multimodal medical analysis and designs a Mixed Fusion structure that supports SHAP-based interpretability. Evaluated on the NDB-UFES dataset, the proposed approach significantly outperforms Transformer-based baselines while maintaining strong interpretability, demonstrating particularly superior performance in sensitivity-oriented metrics such as recall—making it well-suited for high-stakes clinical diagnostic scenarios.
📝 Abstract
This article presents a complementary approach for integrating multimodal medical data in cancer classification, based on state space models represented by the Mamba architecture. To this end, a mixed multimodal fusion architecture, called Mixed Fusion, was employed and developed to enhance the interpretability of the decision-making process. The proposed approach explores two variants of Mamba: one dedicated to visual processing, responsible for classifying the lesion image and generating probabilities associated with the target classes, and another focused on tabular processing, which uses these probabilities together with clinical and/or sociodemographic data to produce the final diagnosis. The experiments were conducted on two medical datasets: PAD-UFES-20, composed of clinical images and information associated with skin lesions, and NDB-UFES, consisting of histopathological images and sociodemographic data related to oral cancer. The results indicate slightly lower performance in balanced accuracy, compared with Transformer-based approaches, on PAD-UFES-20, and superior performance on NDB-UFES. Additionally, substantial gains were observed in the recall metric. Furthermore, the adoption of the Mixed Fusion architecture enables the application of the Shapley Additive Explanations (SHAP) method, increasing the interpretability of the results. These findings indicate that Mamba-based models constitute a suitable alternative for multimodal classification in medical data, especially in scenarios in which sensitivity is a relevant requirement.