Re-M3Dr: Rebalanced MultiModal Mean Deviation Regression

2026-05-25

AI Summary

This work addresses the underperformance of existing multimodal fusion approaches in visual field defect assessment, which stems from distributional discrepancies across modalities and conflicting learning objectives that often result in inferior performance compared to single-modality methods. The study is the first to identify and tackle the coupled imbalance problem inherent in ophthalmic multimodal regression. To this end, it proposes a novel fusion framework that enhances unimodal representations through adaptive margin-supervised contrastive learning and introduces a sharpness-aware gradient modulation mechanism to stabilize joint optimization. Evaluated on both public and private clinical datasets, the proposed method consistently outperforms state-of-the-art multimodal approaches, achieving an average 29% reduction in mean squared error and demonstrating efficient, robust multimodal integration.

Abstract

Mean Deviation (MD) is a critical metric for assessing visual field loss in ophthalmology. While previous work has focused solely on predicting MD from Optical Coherence Tomography (OCT), it is intuitive to assume that combining OCT with another imaging of fundus photography (FP) could improve performance, as two ophthalmic medical imaging provide complementary information. This is particularly expected when sophisticated multi-objective optimization is applied, as documented in common multimodal classification. Surprisingly, our investigations reveal that multimodal fusion in this medical imaging scenario performs worse than unimodal model. Through detailed analysis, we identify the root cause as a coupled imbalance between data distribution and modality learning conflict. This imbalance distorts the optimization landscape, leading to unstable training. To address this challenge, we propose the method of Rebalanced MultiModal Mean Deviation Regression (Re-M3Dr), a novel multimodal regression framework. We enhance unimodal representation through adaptive margin based supervised contrastive learning. Then, our framework stabilizes the joint optimization with the sharpness-aware gradient modulation. Experimental results on both public and private clinical datasets show average 29\% reduction in MSE compared to SOTA multimodal learning methods, demonstrating the superiority of Re-M3Dr. The code is available in the supplementary materials.

Problem

• Mean Deviation
• Multimodal Fusion
• Modality Imbalance
• Medical Imaging Regression
• Optimization Instability

Innovation

• multimodal regression
• modality imbalance
• supervised contrastive learning
• sharpness-aware optimization
• mean deviation prediction