VisioPhysioENet: Multimodal Engagement Detection using Visual and Physiological Signals

2024-09-24

arXiv.org

AI Summary

This study addresses the challenge of fine-grained student attention recognition in classroom settings. We propose VisioPhysioENet, the first dual-level vision–physiology collaborative modeling framework: the vision module extracts dynamic facial keypoint and region-of-interest (ROI) features using Dlib and OpenCV; the physiology module innovatively employs a skin-plane orthogonal method for contactless cardiovascular activity extraction. Evaluated on the DAiSEE dataset, VisioPhysioENet achieves 63.09% accuracy in multi-level attention classification—outperforming the sole existing multimodal baseline by 8.6 percentage points and significantly surpassing state-of-the-art unimodal approaches. This work provides the first empirical validation that cross-modal physiological signals enhance multi-level attention discrimination, establishing a novel paradigm for unobtrusive, fine-grained learning state assessment.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

This paper presents VisioPhysioENet, a novel multimodal system that leverages visual and physiological signals to detect learner engagement. It employs a two-level approach for extracting both visual and physiological features. For visual feature extraction, Dlib is used to detect facial landmarks, while OpenCV provides additional estimations. The face recognition library, built on Dlib, is used to identify the facial region of interest specifically for physiological signal extraction. Physiological signals are then extracted using the plane-orthogonal-toskin method to assess cardiovascular activity. These features are integrated using advanced machine learning classifiers, enhancing the detection of various levels of engagement. We thoroughly tested VisioPhysioENet on the DAiSEE dataset. It achieved an accuracy of 63.09%. This shows it can better identify different levels of engagement compared to many existing methods. It performed 8.6% better than the only other model that uses both physiological and visual features.

Problem

• Student Engagement
• Class Participation
• Attention Level

Innovation

• VisioPhysioENet
• Multimodal Engagement Assessment
• Machine Learning Integration