PhysDrive: A Multimodal Remote Physiological Measurement Dataset for In-vehicle Driver Monitoring

2025-07-25

AI Summary

Existing remote physiological monitoring (RPM) datasets suffer from limited scale, single-modality acquisition, sparse biophysiological annotations, and insufficient real-world driving scenario coverage—hindering robust, contactless physiological monitoring research in automotive environments. To address this, we introduce PhysDrive, the first large-scale, multimodal RPM dataset specifically designed for in-vehicle, contactless sensing. It comprises synchronized RGB, near-infrared, millimeter-wave radar, and six physiological signals—including heart rate, respiration rate, and blood oxygen saturation—collected from 48 drivers across diverse vehicles, lighting conditions, and road scenarios. PhysDrive is the first to systematically integrate heterogeneous sensor modalities with driving-state variables, filling a critical gap in real-world driving RPM data. We further establish an open-source benchmarking framework and a modular codebase compatible with mainstream deep learning frameworks, enabling reproducible cross-modal algorithm development and evaluation. This work significantly advances intelligent cabin systems and non-contact driver state monitoring technologies.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Application Domains: Mobility, Driving & Flight
• Computer Vision: Multi-modal Vision

Application Category

• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Robust and unobtrusive in-vehicle physiological monitoring is crucial for ensuring driving safety and user experience. While remote physiological measurement (RPM) offers a promising non-invasive solution, its translation to real-world driving scenarios is critically constrained by the scarcity of comprehensive datasets. Existing resources are often limited in scale, modality diversity, the breadth of biometric annotations, and the range of captured conditions, thereby omitting inherent real-world challenges in driving. Here, we present PhysDrive, the first large-scale multimodal dataset for contactless in-vehicle physiological sensing with dedicated consideration on various modality settings and driving factors. PhysDrive collects data from 48 drivers, including synchronized RGB, near-infrared camera, and raw mmWave radar data, accompanied with six synchronized ground truths (ECG, BVP, Respiration, HR, RR, and SpO2). It covers a wide spectrum of naturalistic driving conditions, including driver motions, dynamic natural light, vehicle types, and road conditions. We extensively evaluate both signal-processing and deep-learning methods on PhysDrive, establishing a comprehensive benchmark across all modalities, and release full open-source code with compatibility for mainstream public toolboxes. We envision PhysDrive will serve as a foundational resource and accelerate research on multimodal driver monitoring and smart-cockpit systems.

Problem

• Lack of comprehensive datasets for in-vehicle RPM
• Limited modality diversity and biometric annotations in existing datasets
• Challenges in real-world driving conditions for physiological monitoring

Innovation

• Large-scale multimodal dataset for in-vehicle RPM
• Synchronized RGB, NIR, mmWave radar, and biometrics
• Open-source benchmark for signal and deep-learning methods