Occlusion-Robust Multimodal Emotion Recognition in VR via Fusion of Facial Images and EMG

2026-09-03

AI Summary

为解决VR中因头戴设备遮挡上半脸导致的情绪识别问题，通过融合下半脸视频与面部肌电图数据来分类七种情绪，提高识别准确性。

Abstract

Head-mounted displays (HMDs) fundamentally limit emotion recognition in virtual reality (VR): by occluding the upper face, they render conventional image-based facial expression analysis incomplete, particularly for applications requiring real-time affective assessment. We address this challenge by fusing lower-face video with facial electromyography (EMG) from the occluded upper face to classify seven emotional categories (six basic emotions plus neutral). We introduce a synchronized multimodal dataset from 20 participants, pairing lower-face video with seven-channel upper-face EMG elicited by validated emotion stimuli. Under subject-independent test, our proposed late-fusion architecture merging convolutional visual embeddings with RBF-kernel EMG representations achieves 51% macro-F1, outperforming both image-only (41%) and EMG-only (43%) baselines. These results demonstrate that upper-face EMG provides robust complementary information under HMD-induced visual occlusion and establish a foundation for multimodal emotion recognition in naturalistic VR environments. This approach facilitates affect-adaptive applications, including communication training and therapeutic interventions. The dataset will be shared upon request under an ethical-use agreement.

Problem

• Occlusion
• Facial Expression Analysis
• Virtual Reality
• Emotion Recognition
• Head-mounted Displays

Innovation

• multimodal emotion recognition
• facial electromyography (EMG)
• late-fusion architecture
• virtual reality (VR)