Fusion-E2Pulse: A Multimodal Event-RGB Fusion Network for Non-contact Pulse Wave Reconstruction

2026-06-14

AI Summary

This work addresses the limitations of conventional RGB-based methods, which often lose subtle pulse wave details due to camera integration effects, and event cameras, which, despite their high temporal sensitivity, are prone to motion artifacts. The study proposes the first multimodal framework that synergistically combines RGB video and event camera data for non-contact pulse wave reconstruction. Specifically, filtered RGB signals provide structural priors to suppress motion-induced artifacts, while the high temporal resolution of event streams enables recovery of fine waveform features—such as the dicrotic notch. The proposed fusion network achieves state-of-the-art performance in heart rate estimation (MAE = 0.78 bpm), waveform fidelity (correlation coefficient = 0.89), and systolic phase duration accuracy (error = 16.74 ms), demonstrating robustness to noise and strong capability in preserving clinically relevant morphological characteristics.

Abstract

Non-contact pulse wave reconstruction hinges on the precise recovery of waveform morphology, including the dicrotic notch. Conventional Red-Green-Blue (RGB)-based methods, which extract physiological signals from recorded facial videos, are constrained by the integral imaging mechanism of standard cameras, where the exposure process induces a smoothing effect that attenuates subtle vascular pulsation details. Conversely, neuromorphic event cameras, while offering exceptional sensitivity to intensity fluctuations, are inherently susceptible to noise and artifacts induced by minor motion. To exploit the synergy between frame-based integration and event-based differential sensing, we propose a novel multimodal network named Fusion-E2Pulse. This framework utilizes filtered RGB signals as structural priors to suppress motion artifacts, while leveraging the high-sensitivity of event streams to recover fine-grained morphological details. Experimental results demonstrate that Fusion-E2Pulse achieves state-of-the-art performance, effectively balancing noise suppression and morphological fidelity, achieving a mean absolute error of 0.78 bpm for heart rate estimation, a waveform correlation of 0.89, and a systolic phase duration error of 16.74 ms, validating its efficacy in reconstructing fine-grained pathological features.

Problem

• non-contact pulse wave reconstruction
• RGB-based methods
• event cameras
• motion artifacts
• waveform morphology

Innovation

• multimodal fusion
• event camera
• non-contact pulse wave reconstruction
• morphological fidelity
• RGB-event synergy