CardiacMamba: A Multimodal RGB-RF Fusion Framework with State Space Models for Remote Physiological Measurement

2025-02-19

AI Summary

To address the accuracy–robustness trade-off in remote photoplethysmography (rPPG)-based heart rate (HR) estimation—caused by illumination variations, motion artifacts, and skin-tone bias—this paper proposes CardiacMamba, the first framework integrating RGB and radio-frequency (RF) dual-modal signals. Methodologically: (1) a Temporal-Difference Mamba Module (TDMM) is designed to model RF signal dynamics; (2) a bidirectional state-space model enables cross-modal temporal alignment; and (3) a channel-wise Fast Fourier Transform (CFFT) module enhances frequency-domain feature extraction and fusion. Evaluated on the EquiPleth dataset, CardiacMamba achieves state-of-the-art performance: significantly reduced HR estimation error, more reliable periodicity detection, markedly alleviated skin-tone bias, and strong robustness under single-modal dropout. These advances enhance the practical deployment of contactless rPPG in clinical and health-monitoring applications.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Heart rate (HR) estimation via remote photoplethysmography (rPPG) offers a non-invasive solution for health monitoring. However, traditional single-modality approaches (RGB or Radio Frequency (RF)) face challenges in balancing robustness and accuracy due to lighting variations, motion artifacts, and skin tone bias. In this paper, we propose CardiacMamba, a multimodal RGB-RF fusion framework that leverages the complementary strengths of both modalities. It introduces the Temporal Difference Mamba Module (TDMM) to capture dynamic changes in RF signals using timing differences between frames, enhancing the extraction of local and global features. Additionally, CardiacMamba employs a Bidirectional SSM for cross-modal alignment and a Channel-wise Fast Fourier Transform (CFFT) to effectively capture and refine the frequency domain characteristics of RGB and RF signals, ultimately improving heart rate estimation accuracy and periodicity detection. Extensive experiments on the EquiPleth dataset demonstrate state-of-the-art performance, achieving marked improvements in accuracy and robustness. CardiacMamba significantly mitigates skin tone bias, reducing performance disparities across demographic groups, and maintains resilience under missing-modality scenarios. By addressing critical challenges in fairness, adaptability, and precision, the framework advances rPPG technology toward reliable real-world deployment in healthcare. The codes are available at: https://github.com/WuZheng42/CardiacMamba.

Problem

• Enhance heart rate estimation accuracy
• Mitigate skin tone bias in rPPG
• Improve robustness in missing-modality scenarios

Innovation

• Multimodal RGB-RF fusion
• Temporal Difference Mamba Module
• Bidirectional SSM alignment