Human Centered Non Intrusive Driver State Modeling Using Personalized Physiological Signals in Real World Automated Driving

2026-04-13

AI Summary

This study addresses the limitations of existing driver monitoring systems, which predominantly rely on generic models that overlook individual physiological differences and thus struggle to reliably detect driver states in real-world Level 2 autonomous driving scenarios. To overcome this, the authors present the first in-vehicle investigation leveraging the Empatica E4 wearable device to collect multimodal physiological signals—such as electrodermal activity and heart rate—during actual road driving. These signals are transformed into two-dimensional image representations and fed into a personalized deep learning framework based on a pre-trained ResNet50 architecture. Experimental results demonstrate that the personalized model achieves an average accuracy of 92.68%, substantially outperforming cross-user generic models, which attain only 54% accuracy. This significant improvement underscores the critical importance of tailoring models to individual physiological characteristics for enhancing driver state recognition performance.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Application Domains: Mobility, Driving & Flight

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Large-scale security measurements

Abstract

In vehicles with partial or conditional driving automation (SAE Levels 2-3), the driver remains responsible for supervising the system and responding to take-over requests. Therefore, reliable driver monitoring is essential for safe human-automation collaboration. However, most existing Driver Monitoring Systems rely on generalized models that ignore individual physiological variability. In this study, we examine the feasibility of personalized driver state modeling using non-intrusive physiological sensing during real-world automated driving. We conducted experiments in an SAE Level 2 vehicle using an Empatica E4 wearable sensor to capture multimodal physiological signals, including electrodermal activity, heart rate, temperature, and motion data. To leverage deep learning architectures designed for images, we transformed the physiological signals into two-dimensional representations and processed them using a multimodal architecture based on pre-trained ResNet50 feature extractors. Experiments across four drivers demonstrate substantial interindividual variability in physiological patterns related to driver awareness. Personalized models achieved an average accuracy of 92.68%, whereas generalized models trained on multiple users dropped to an accuracy of 54%, revealing substantial limitations in cross-user generalization. These results underscore the necessity of adaptive, personalized driver monitoring systems for future automated vehicles and imply that autonomous systems should adapt to each driver's unique physiological profile.

Problem

• driver monitoring
• personalized modeling
• physiological signals
• automated driving
• interindividual variability

Innovation

• personalized modeling
• non-intrusive physiological sensing
• driver state monitoring
• multimodal deep learning
• real-world automated driving