FMT:A Multimodal Pneumonia Detection Model Based on Stacking MOE Framework
To address the poor robustness of clinical pneumonia diagnosis models under incomplete or missing multimodal data, this paper proposes a flexible multimodal detection framework. First, a dynamic masking attention mechanism is designed to explicitly model stochastic modality dropout across imaging and textual modalities. Second, a sequential Mixture-of-Experts (MoE) architecture is introduced to enable hierarchical cross-modal feature fusion and decision refinement. Third, ResNet-50 and BERT are jointly fine-tuned for cross-modal representation learning. Evaluated on a small-sample multimodal pneumonia dataset, the framework achieves 94% accuracy, 95% recall, and 93% F1-score—significantly outperforming unimodal baselines and state-of-the-art methods such as CheXMed. These results demonstrate superior generalization under modality missingness and strong clinical applicability.