RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation

2026-04-14

AI Summary

This work addresses the limitation of existing cross-modal fusion methods, which typically assume all modalities are equally reliable and consequently suffer significant performance degradation when auxiliary modalities contain noise, misalignment, or missing data. To overcome this, the authors propose a reliability-aware, self-gated State Space Model (Mamba) that dynamically selects and aggregates trustworthy features through a self-gating mechanism. Additionally, a lightweight local cross-gating modulation is introduced to enhance fine-grained detail modeling, enabling efficient fusion of both global and local multimodal features. The proposed method achieves state-of-the-art performance across multiple benchmarks—including NYUDepth V2, SUN-RGBD, MFNet, and PST900—with a maximum mIoU improvement of 1.6% while using only 48.6 million parameters.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multimodal semantic segmentation has emerged as a powerful paradigm for enhancing scene understanding by leveraging complementary information from multiple sensing modalities (e.g., RGB, depth, and thermal). However, existing cross-modal fusion methods often implicitly assume that all modalities are equally reliable, which can lead to feature degradation when auxiliary modalities are noisy, misaligned, or incomplete. In this paper, we revisit cross-modal fusion from the perspective of modality reliability and propose a novel framework termed the Reliability-aware Self-Gated State Space Model (RSGMamba). At the core of our method is the Reliability-aware Self-Gated Mamba Block (RSGMB), which explicitly models modality reliability and dynamically regulates cross-modal interactions through a self-gating mechanism. Unlike conventional fusion strategies that indiscriminately exchange information across modalities, RSGMB enables reliability-aware feature selection and enhancing informative feature aggregation. In addition, a lightweight Local Cross-Gated Modulation (LCGM) is incorporated to refine fine-grained spatial details, complementing the global modeling capability of RSGMB. Extensive experiments demonstrate that RSGMamba achieves state-of-the-art performance on both RGB-D and RGB-T semantic segmentation benchmarks, resulting 58.8% / 54.0% mIoU on NYUDepth V2 and SUN-RGBD (+0.4% / +0.7% over prior best), and 61.1% / 88.9% mIoU on MFNet and PST900 (up to +1.6%), with only 48.6M parameters, thereby validating the effectiveness and superiority of the proposed approach.

Problem

• multimodal semantic segmentation
• cross-modal fusion
• modality reliability
• feature degradation
• noisy modalities

Innovation

• Reliability-aware fusion
• Self-gated mechanism
• State Space Model
• Multimodal semantic segmentation
• Cross-modal interaction