UMSS: Towards Unsupervised Multi-modal Semantic Segmentation

2026-07-14

AI Summary

This work addresses the degradation of modality fusion in unsupervised multimodal semantic segmentation caused by the absence of labeled data. To this end, it proposes UniM2, the first framework that constructs a unified latent space under fully unsupervised conditions to enable cross-modal semantic alignment and fusion. Built upon the DINOv3 architecture, UniM2 introduces two key innovations: a Cross-Modal Correspondence Synergy (CMCS) mechanism and an RGB-referenced Cross-Modal Harmonizer (CMH), which together facilitate adaptive, label-free fusion of multimodal features. Evaluated on the NYU Depth v2 and MFNet datasets, the method achieves significant improvements of 6.4% and 9.8% in mIoU, respectively, substantially outperforming existing unsupervised approaches.

Abstract

Multimodal semantic segmentation (MSS) is essential for robust perception in complex environments, yet its potential remains largely untapped because of the prohibitive cost of human annotations. While unsupervised semantic segmentation (USS) has achieved strong results on a single RGB modality, its naive extension to multimodal data is often hindered by fusion degradation. This occurs because, without explicit supervision, existing frameworks struggle to reconcile the heterogeneous structural patterns captured by different sensors and therefore fail to effectively exploit their complementary information. In this paper, we make the first attempt to address the novel problem of Unsupervised Multimodal Semantic Segmentation (UMSS), aiming to effectively exploit complementary sensor information in a fully label free setting. To this end, we propose UniM2 (Unified Multimodal), a novel framework built on DINOv3 that transforms conventional fusion methods into consistent performance gains. Our key idea is to learn a unified latent space driven by Cross Modal Correspondence Synergy (CMCS) to extract intrinsic shared semantic cues, bypassing the need for label guided adaptive fusion. To mitigate inherent intermodal conflicts, we introduce a Cross Modal Harmonizer (CMH) that designates RGB as a stable reference, effectively suppressing inconsistent relational supervision while guiding the model to exploit complementary structural features. Extensive experimental results on NYU Depth v2 and MFNet show that UniM2 improves mIoU by 6.4% and 9.8%, respectively, demonstrating clear advantages over existing frameworks for UMSS.

Problem

• Unsupervised Multimodal Semantic Segmentation
• Multimodal Fusion
• Semantic Segmentation
• Cross-modal Learning
• Label-free Perception

Innovation

• Unsupervised Multimodal Semantic Segmentation
• Cross Modal Correspondence Synergy
• Cross Modal Harmonizer
• Unified Latent Space
• Label-free Fusion