Seismocardiography for Emotion Recognition: A Study on EmoWear with Insights from DEAP

2024-11-30

arXiv.org

AI Summary

This work addresses the strong dependency of traditional emotion recognition (ER) systems on physiological sensors—such as electrocardiography (ECG) and blood volume pulse (BVP)—which hinder seamless daily deployment. We propose, for the first time, a novel single-device ER paradigm leveraging seismocardiography (SCG) and acceleration-derived respiration (ADR) as independent, complementary modalities. Using only a chest-worn accelerometer, SCG and ADR signals are simultaneously acquired. We design an end-to-end deep learning framework (CNN-LSTM) incorporating time-frequency features and single-trial classification to decode valence–arousal dimensions on EmoWear and DEAP datasets. Results demonstrate that SCG alone achieves performance comparable to ECG/BVP; moreover, SCG+ADR fusion exhibits superior robustness under realistic wear conditions, significantly enhancing the feasibility and practicality of everyday affective computing.

Technology Category

• Cognitive Modeling & Cognitive Systems: Affective Computing
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Biometrics, Face, Gesture & Pose

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Emotions have a profound impact on our daily lives, influencing our thoughts, behaviors, and interactions, but also our physiological reactions. Recent advances in wearable technology have facilitated studying emotions through cardio-respiratory signals. Accelerometers offer a non-invasive, convenient, and cost-effective method for capturing heart- and pulmonary-induced vibrations on the chest wall, specifically Seismocardiography (SCG) and Accelerometry-Derived Respiration (ADR). Their affordability, wide availability, and ability to provide rich contextual data make accelerometers ideal for everyday use. While accelerometers have been used as part of broader modality fusions for Emotion Recognition (ER), their stand-alone potential via SCG and ADR remains unexplored. Bridging this gap could significantly help the embedding of ER into real-world applications. To address this gap, we introduce SCG as a novel modality for ER and evaluate its performance using the EmoWear dataset. First, we replicate the single-trial emotion classification pipeline from the DEAP dataset study, achieving similar results. Then we use our validated pipeline to train models that predict affective valence-arousal states using SCG and compare them against established cardiac signals, Electrocardiography (ECG) and Blood Volume Pulse (BVP). Results show that SCG is a viable modality for ER, achieving similar performance to ECG and BVP. By combining ADR with SCG, we achieved a working ER framework that only requires a single chest-worn accelerometer. These findings pave the way for integrating ER into real-world, enabling seamless affective computing in everyday life.

Problem

• Explores Seismocardiography (SCG) for emotion recognition using wearable technology.
• Compares SCG performance with ECG and BVP for affective state prediction.
• Develops a single accelerometer-based framework for real-world emotion recognition.

Innovation

• Seismocardiography (SCG) for emotion recognition
• Single chest-worn accelerometer for SCG and ADR
• SCG achieves comparable performance to ECG and BVP