Multimodal Semantic-Aware Contrastive Learning For False Negative Mitigation in 3D Medical Imaging

2026-07-16

AI Summary

This work addresses a critical limitation in conventional multimodal contrastive learning for medical imaging, where semantically similar but unpaired samples are erroneously treated as negative pairs, degrading representation quality. To mitigate this issue, the authors propose a novel semantic-aware multimodal contrastive learning framework that leverages the semantic similarity among radiology reports as a guiding signal to align 3D brain MRI scans with their corresponding textual descriptions. By integrating multimodal contrastive learning, semantic similarity modeling, 3D medical image analysis, and natural language processing of radiology reports, the method effectively alleviates the false-negative problem. Evaluated on pediatric brain tumor molecular subtyping, the approach demonstrates substantial performance gains, achieving at least a 22.6% improvement in AUC after pretraining and significantly enhancing downstream task efficacy.

Abstract

Multimodal Contrastive Learning (CL) has shown significant performance in aligning representations across various data modalities and improving downstream tasks, especially in healthcare. It works by minimizing the distance between matched (positive) data modalities, while maximizing the distance between mismatched (negative) samples. Traditional CL frameworks typically assume instance-based correspondence within data batches, treating all non-paired samples as negatives. However, this assumption often fails in medical settings, where samples may share high-level semantic attributes, leading to false negatives that degrade representation quality. In this paper, we propose Multimodal Semantic-Aware Contrastive Learning (MseaCL), a CL framework trained on a pediatric cohort of 3D brain magnetic resonance imaging (MRI) scans and radiology reports. The goal of this framework is to mitigate the impact of semantically similar false negative samples by incorporating semantic similarity between radiology reports, as a guiding signal during the learning process. Our results indicate that applying this framework as a pretraining stage can achieve notable improvements in downstream tasks, e.g., at least a 22.6\% increase in the area under the receiver operating characteristic curve (AUC) of pediatric brain tumor molecular classification, demonstrating its potential for more robust and semantically aligned multimodal representations in clinical applications.

Problem

• False Negative
• Multimodal Contrastive Learning
• 3D Medical Imaging
• Semantic Similarity
• Representation Learning

Innovation

• Multimodal Contrastive Learning
• False Negative Mitigation
• Semantic-Aware Learning
• 3D Medical Imaging
• Radiology Report