MUPA$^{2}$E: Multimodal Unified Perception with Asymmetric Attention for Emotion Assessment

2026-08-16

AI Summary

This study addresses the challenges of disjointed feature extraction and inefficient fusion in multimodal emotion assessment by proposing a unified perception framework based on a shared asymmetric attention backbone. The method efficiently integrates facial video and EEG signals through axis-folded frame tokens and spatial EEG projection, while employing a duration cropping strategy to mitigate data distribution bias. Experimental results demonstrate that this compact architecture achieves an accuracy of 62.71% after controlling for duration bias and 70.07% on raw test sets. These findings validate the feasibility of utilizing a single shared architecture to process heterogeneous signals, effectively resolving critical issues regarding fusion efficiency and classification cue bias in multimodal emotion recognition.

Abstract

Automatic emotion assessment can benefit from combining neural and behavioral signals, but many multimodal approaches rely on separate, modality-specific feature-extraction pipelines before fusion. This paper presents MUPA\textsuperscript{2}E, a unified perception framework that processes facial video and electroencephalography (EEG) through a single shared asymmetric-attention backbone. Facial video is represented through axis-folded frame tokens, while EEG is processed either as a raw multichannel waveform or projected into the spatial domain for multimodal fusion. The framework is evaluated on the DMER dataset under a stratified subject-independent protocol, comparing unimodal video, unimodal EEG, and fused video--EEG configurations with per-channel and merged EEG projections. Using the original recordings, with shorter trials zero-padded to match the longest duration, merged fusion at stride~$30$ achieves the highest validation performance and a test accuracy of $70.07\%$. Further analysis revealed that recording duration is unevenly distributed across the affective classes, making the padding pattern a potential classification cue. Controlling for this factor by cropping all recordings to a common duration of $20$ seconds yielded a test accuracy of $62.71\%$, providing a stricter duration-controlled assessment of the framework in which differences in recording length are removed as a potential classification cue. These findings demonstrate the feasibility of processing structurally different neural and visual signals within a compact unified architecture while highlighting the importance of controlling duration-related cues in affective datasets.

Problem

• Multimodal Emotion Assessment
• Unified Perception
• EEG
• Facial Video
• Duration Cue

Innovation

• Unified Perception Framework
• Asymmetric Attention
• Multimodal Fusion
• EEG-Video Integration
• Duration-controlled Assessment