UL-DD: A Multimodal Drowsiness Dataset Using Video, Biometric Signals, and Behavioral Data

2025-07-16

AI Summary

Existing driver fatigue detection research is hindered by the absence of a continuous, multimodal public dataset integrating facial, behavioral, and physiological signals. To address this gap, we introduce the first open multimodal dataset explicitly designed for progressive fatigue modeling. It synchronously captures 3D facial video (depth + infrared), heart rate, skin conductance response, blood oxygen saturation, body temperature, triaxial acceleration, and driving behavior signals—time-aligned with truck simulator telemetry and Karolinska Sleepiness Scale (KSS) annotations. Crucially, it provides fine-grained, continuous fatigue labels at second-level resolution over 40-minute sessions, departing from conventional discrete-label paradigms. The dataset comprises 1,400 minutes of high-quality recordings from 19 participants. It enables rigorous investigation of fatigue dynamics and facilitates benchmarking of cross-modal fusion algorithms for real-world driver monitoring systems.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

In this study, we present a comprehensive public dataset for driver drowsiness detection, integrating multimodal signals of facial, behavioral, and biometric indicators. Our dataset includes 3D facial video using a depth camera, IR camera footage, posterior videos, and biometric signals such as heart rate, electrodermal activity, blood oxygen saturation, skin temperature, and accelerometer data. This data set provides grip sensor data from the steering wheel and telemetry data from the American truck simulator game to provide more information about drivers' behavior while they are alert and drowsy. Drowsiness levels were self-reported every four minutes using the Karolinska Sleepiness Scale (KSS). The simulation environment consists of three monitor setups, and the driving condition is completely like a car. Data were collected from 19 subjects (15 M, 4 F) in two conditions: when they were fully alert and when they exhibited signs of sleepiness. Unlike other datasets, our multimodal dataset has a continuous duration of 40 minutes for each data collection session per subject, contributing to a total length of 1,400 minutes, and we recorded gradual changes in the driver state rather than discrete alert/drowsy labels. This study aims to create a comprehensive multimodal dataset of driver drowsiness that captures a wider range of physiological, behavioral, and driving-related signals. The dataset will be available upon request to the corresponding author.

Problem

• Creating a multimodal dataset for driver drowsiness detection
• Integrating facial, behavioral, and biometric signals for analysis
• Capturing gradual state changes in drivers over time

Innovation

• Multimodal dataset with video and biometric signals
• Includes grip sensor and telemetry behavioral data
• Continuous 40-minute sessions capture gradual drowsiness changes