Multimodal Fusion for Fine-Grained Classification of Breast Fibroadenoma and Phyllodes Tumors

2026-07-02

AI Summary

This study addresses the diagnostic challenge posed by the high ultrasonographic similarity between fibroadenomas and phyllodes tumors, which frequently leads to preoperative misdiagnosis. To tackle this issue, the authors introduce FAPT-M, the first multimodal dataset integrating ultrasound images, clinical attributes, and diagnostic text reports. They propose a novel multimodal classification framework that leverages a clinical-guided adaptive modulation mechanism and dual-path representation learning to effectively align and fuse visual features (extracted via DenseNet), textual embeddings (inspired by CLIP), and lightweight clinical encodings. Evaluated under five-fold cross-validation, the model achieves 77.64% accuracy, 73.38% F1 score, and 89.74% AUC, significantly outperforming existing unimodal and multimodal baselines, thereby establishing a new benchmark for multimodal analysis in breast ultrasound diagnostics.

Abstract

Breast fibroadenoma (FA) and phyllodes tumor (PT) are fibroepithelial breast lesions with highly overlapping appearances on B-mode ultrasound, making benign and borderline PT prone to being misclassified as FA and complicating preoperative decision-making. Existing computer-aided diagnosis methods commonly rely on single-modal imaging features and insufficiently exploit complementary clinical and textual information. To address this limitation, we construct the FAPT-M Dataset, a pathology-confirmed multimodal dataset comprising 910 patients with strictly reviewed ultrasound images, structured clinical attributes, and ultrasound diagnostic descriptions. Based on this dataset, we propose a clinically guided multimodal framework that integrates DenseNet-based visual encoding, CLIP-inspired text encoding, and lightweight clinical encoding, and further introduces clinical-conditioned adaptive modulation, cross-modal Transformer fusion, and dual-path representation learning to improve feature alignment and multimodal interaction. Under patient-level five-fold cross-validation, the proposed method achieves an accuracy of 77.64%, F1-score of 73.38%, and AUC of 89.74%, outperforming representative CNN-, Transformer-, and vision-language-based baselines. Ablation studies and class-balanced evaluations further confirm the contribution of three-modality fusion and the key architectural components. Overall, this work provides an effective multimodal approach for fine-grained FA-PT classification and establishes a high-quality benchmark for multimodal breast ultrasound analysis.

Problem

• multimodal fusion
• fine-grained classification
• breast fibroadenoma
• phyllodes tumor
• computer-aided diagnosis

Innovation

• multimodal fusion
• clinical-conditioned modulation
• cross-modal Transformer
• fine-grained classification
• vision-language modeling