Graph-based Online Monitoring of Train Driver States via Facial and Skeletal Features

2025-05-09

AI Summary

To address the limited vigilance assessment capability of conventional methods (e.g., dead-man switches) in train driver fatigue monitoring, this paper proposes an online multimodal state monitoring system tailored to driving scenarios. Methodologically, it fuses OpenPose-based skeletal and MediaPipe-based facial features, constructs the first driver dataset incorporating simulated pathological states, and designs a customized Directed Graph Neural Network (DGNN) for real-time classification into vigilant, non-vigilant, and pathological states. Key contributions include: (1) the first incorporation of pathological state modeling in driver monitoring; (2) a driving-adapted directed graph architecture encoding spatiotemporal anatomical dependencies; and (3) an ablation-driven multimodal feature optimization strategy. Experiments achieve 80.88% accuracy for three-class classification and 99.12% for binary (vigilant/non-vigilant) classification, with low-latency online deployment feasibility—significantly outperforming unimodal baselines.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Driver fatigue poses a significant challenge to railway safety, with traditional systems like the dead-man switch offering limited and basic alertness checks. This study presents an online behavior-based monitoring system utilizing a customised Directed-Graph Neural Network (DGNN) to classify train driver's states into three categories: alert, not alert, and pathological. To optimize input representations for the model, an ablation study was performed, comparing three feature configurations: skeletal-only, facial-only, and a combination of both. Experimental results show that combining facial and skeletal features yields the highest accuracy (80.88%) in the three-class model, outperforming models using only facial or skeletal features. Furthermore, this combination achieves over 99% accuracy in the binary alertness classification. Additionally, we introduced a novel dataset that, for the first time, incorporates simulated pathological conditions into train driver monitoring, broadening the scope for assessing risks related to fatigue and health. This work represents a step forward in enhancing railway safety through advanced online monitoring using vision-based technologies.

Problem

• Classify train driver states using facial and skeletal features
• Improve accuracy in detecting alertness and pathological conditions
• Enhance railway safety with vision-based online monitoring

Innovation

• Custom DGNN for driver state classification
• Combined facial and skeletal features for accuracy
• Novel dataset with simulated pathological conditions