Balancing Multimodal Learning through Label Space Reshaping

2026-05-22

AI Summary

This work addresses the challenge of modality imbalance in multimodal learning, where disparities in convergence rates often cause faster-converging modalities to dominate optimization while slower ones remain under-trained. To mitigate this issue, the paper introduces Balanced Multimodal Learning via Label-space Reshaping (BMLR), a novel strategy that aligns inter-modal learning difficulties by deliberately adjusting the mapping complexity from each modality to the label space. By reshaping the label space, BMLR facilitates more effective cross-modal interaction and enhances inter-class discriminability without altering the underlying model architecture. Theoretical analysis and extensive experiments demonstrate that BMLR consistently improves performance across diverse mainstream multimodal architectures, highlighting its strong compatibility and effectiveness as a plug-and-play solution for balanced multimodal representation learning.

Abstract

Multimodal learning often suffers from modality imbalance, where modalities that converge faster dominate optimization while others remain undertrained. Existing approaches typically mitigate this issue by strengthening the weak modality or adjusting optimization gradients. However, such strategies mainly compensate for optimization rate discrepancies, often at the expense of the strong modality's optimization capacity, without analyzing how these discrepancies arise at the modality level. Based on theoretical insights and empirical observations, we argue that the discrepancy of learning pace arises from differences in the mapping difficulty between modality-specific feature space and the shared label space. To address this issue, we propose Balanced Multimodal Label Reshaping (BMLR), the first method that promotes multimodal balance from the label-side design. BMLR reshapes the cross-modal label space to equalize mapping difficulty across modalities, thereby facilitating modality interaction and injecting richer inter-class information into each modality. Extensive experiments across multiple architectures demonstrate that BMLR consistently improves multimodal performance and exhibits strong compatibility with diverse model designs. The source code will be released soon.

Problem

• multimodal learning
• modality imbalance
• label space
• learning pace discrepancy
• feature-to-label mapping

Innovation

• multimodal learning
• modality imbalance
• label space reshaping
• mapping difficulty
• BMLR