MS-rPPG: Multi-spectral State Space Model for Remote Photoplethysmography in Driver Monitoring Systems

2026-06-19

AI Summary

This study addresses the significant degradation in heart rate estimation accuracy of existing vision-based remote photoplethysmography (rPPG) methods under uncontrolled driving conditions characterized by illumination variations and head movements. To overcome this challenge, the authors propose MS-rPPG, a novel multispectral framework that, for the first time, integrates RGB and near-infrared (NIR) video streams into an in-vehicle rPPG system. They introduce a frequency-domain cross-spectral linear modulation (CSLM) mechanism to align multispectral signals and develop a new state-space model, MS-Mamba, to jointly capture long-range temporal dependencies and cross-channel interactions within multispectral sequences. Experiments on the MR-NIRP Car benchmark and the newly collected MS-Drive dataset demonstrate that the proposed method substantially outperforms current state-of-the-art approaches, achieving notable improvements in both accuracy and robustness.

Abstract

Remote photoplethysmography (rPPG) is a camera-based technique for measuring physiological signals, particularly cardiac activity. From the remotely measured signals, heart rate can be estimated, which is crucial for health monitoring. In this study, we investigate a driver health monitoring system based on remote heart rate estimation. However, driving environments represent uncontrolled settings where videos are subject to varying illumination conditions and frequent head movements. We introduce MS-rPPG, a multi-spectral framework that combines RGB with near-infrared (NIR) face video to alleviate rPPG estimation under challenging driving conditions. To combine the complementary features from two spectral videos, we propose a cross-spectral linear modulation (CSLM) strategy based on frequency-domain analysis. Moreover, we introduce MS-Mamba, a novel state space model designed to effectively model long-range temporal dependencies while jointly capturing cross-channel interactions between multi-spectral features. We collected a real-world dataset called MS-Drive, which was recorded from 50 participants while driving the vehicle. The proposed method was evaluated on the MR-NIRP Car dataset and MS-Drive datasets. The experimental results indicate that MS-rPPG shows better robustness and heart rate estimation accuracy than previous methods, highlighting its promise for driver health monitoring. The codes are available at github.com/ziiho08/MS-rPPG.

Problem

• remote photoplethysmography
• driver monitoring
• uncontrolled illumination
• head movement
• heart rate estimation

Innovation

• multi-spectral
• remote photoplethysmography
• state space model
• cross-spectral modulation
• driver monitoring