UniMod: Enhancing Multi-Modal Medical Diagnosis through Cross-Modality and Within-Modality Alignment

2026-08-10

AI Summary

This work addresses shortcut learning in multimodal medical diagnosis, where models often over-rely on easily learned modalities like text while overlooking subtle pathological cues in images. To mitigate this issue, the authors propose a plug-and-play multi-task learning framework that enforces independent diagnostic predictions from image and text modalities without altering the model architecture. Cross-modal knowledge transfer and intra-modal supervised contrastive alignment on co-diagnosed samples are leveraged to enhance multimodal synergy. The approach effectively suppresses shortcut learning and substantially improves multi-label diagnostic performance: it achieves an AUC of 0.850 on Harvard-Glaucoma (outperforming OGM-GE and Gradient Blending by 1.6–1.8%) and 0.966 on CheXpert Plus (surpassing prior methods by over 5%), with an average AUC gain of 0.097 across five disease categories.

Abstract

Multi-modal learning combining medical images and clinical text is promising for disease diagnosis. However, standard multi-modal training leads to shortcut learning: models exploit the easier modality (e.g., diagnostic cues in text) while neglecting harder-to-learn features (e.g., subtle visual patterns). We propose UniMod, a framework that mitigates shortcut learning by requiring each modality to predict the diagnosis on its own. It supervises image-only, text-only, and multi-modal classification simultaneously, so each modality must extract diagnostic features. We add cross-modality alignment for knowledge transfer and within-modality supervised contrastive alignment over same-diagnosis patients. On Harvard-Glaucoma, UniMod reaches 0.850 AUC, outperforming OGM-GE and Gradient Blending by 1.6-1.8%; on CheXpert Plus, it reaches 0.966 AUC, surpassing them by over 5%. UniMod also extends to 5-class multi-label diagnosis without architectural change, improving mean AUC by 0.097 over CGGM.

Problem

• multi-modal learning
• shortcut learning
• medical diagnosis
• cross-modality alignment
• within-modality alignment

Innovation

• shortcut learning
• cross-modality alignment
• within-modality contrastive alignment
• multi-modal medical diagnosis
• UniMod