🤖 AI Summary
To address the poor robustness of clinical pneumonia diagnosis models under incomplete or missing multimodal data, this paper proposes a flexible multimodal detection framework. First, a dynamic masking attention mechanism is designed to explicitly model stochastic modality dropout across imaging and textual modalities. Second, a sequential Mixture-of-Experts (MoE) architecture is introduced to enable hierarchical cross-modal feature fusion and decision refinement. Third, ResNet-50 and BERT are jointly fine-tuned for cross-modal representation learning. Evaluated on a small-sample multimodal pneumonia dataset, the framework achieves 94% accuracy, 95% recall, and 93% F1-score—significantly outperforming unimodal baselines and state-of-the-art methods such as CheXMed. These results demonstrate superior generalization under modality missingness and strong clinical applicability.
📝 Abstract
Artificial intelligence has shown the potential to improve diagnostic accuracy through medical image analysis for pneumonia diagnosis. However, traditional multimodal approaches often fail to address real-world challenges such as incomplete data and modality loss. In this study, a Flexible Multimodal Transformer (FMT) was proposed, which uses ResNet-50 and BERT for joint representation learning, followed by a dynamic masked attention strategy that simulates clinical modality loss to improve robustness; finally, a sequential mixture of experts (MOE) architecture was used to achieve multi-level decision refinement. After evaluation on a small multimodal pneumonia dataset, FMT achieved state-of-the-art performance with 94% accuracy, 95% recall, and 93% F1 score, outperforming single-modal baselines (ResNet: 89%; BERT: 79%) and the medical benchmark CheXMed (90%), providing a scalable solution for multimodal diagnosis of pneumonia in resource-constrained medical settings.