SGMA: Semantic-Guided Modality-Aware Segmentation for Remote Sensing with Incomplete Multimodal Data

2026-03-02

AI Summary

Remote sensing multimodal semantic segmentation often suffers from missing modalities due to sensor failures or incomplete coverage, encountering three major challenges: modality imbalance, large intra-class variation, and cross-modal heterogeneity. To address these issues, this work proposes a Semantic-Guided Modality-Aware framework (SGMA), comprising two plug-and-play modules—Semantic-Guided Fusion (SGF) and Modality-Aware Sampling (MAS). SGMA leverages multi-scale class semantic prototypes, prototype-feature alignment for modality robustness assessment, adaptive weighted fusion, and dynamic sample reweighting to jointly mitigate intra-class variability and cross-modal inconsistency while balancing modality contributions. Extensive experiments demonstrate that SGMA significantly outperforms existing methods across multiple remote sensing datasets and backbone networks, with particularly notable performance gains under fragile or degraded modalities.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multimodal semantic segmentation integrates complementary information from diverse sensors for remote sensing Earth observation. However, practical systems often encounter missing modalities due to sensor failures or incomplete coverage, termed Incomplete Multimodal Semantic Segmentation (IMSS). IMSS faces three key challenges: (1) multimodal imbalance, where dominant modalities suppress fragile ones; (2) intra-class variation in scale, shape, and orientation across modalities; and (3) cross-modal heterogeneity with conflicting cues producing inconsistent semantic responses. Existing methods rely on contrastive learning or joint optimization, which risk over-alignment, discarding modality-specific cues or imbalanced training, favoring robust modalities, while largely overlooking intra-class variation and cross-modal heterogeneity. To address these limitations, we propose the Semantic-Guided Modality-Aware (SGMA) framework, which ensures balanced multimodal learning while reducing intra-class variation and reconciling cross-modal inconsistencies through semantic guidance. SGMA introduces two complementary plug-and-play modules: (1) Semantic-Guided Fusion (SGF) module extracts multi-scale, class-wise semantic prototypes that capture consistent categorical representations across modalities, estimates per-modality robustness based on prototype-feature alignment, and performs adaptive fusion weighted by robustness scores to mitigate intra-class variation and cross-modal heterogeneity; (2) Modality-Aware Sampling (MAS) module leverages robustness estimations from SGF to dynamically reweight training samples, prioritizing challenging samples from fragile modalities to address modality imbalance. Extensive experiments across multiple datasets and backbones demonstrate that SGMA consistently outperforms state-of-the-art methods, with particularly significant improvements in fragile modalities.

Problem

• Incomplete Multimodal Semantic Segmentation
• Multimodal Imbalance
• Intra-class Variation
• Cross-modal Heterogeneity
• Remote Sensing

Innovation

• Semantic-Guided Fusion
• Modality-Aware Sampling
• Incomplete Multimodal Semantic Segmentation
• Cross-Modal Heterogeneity
• Intra-Class Variation