HADUA: Hierarchical Attention and Dynamic Uniform Alignment for Robust Cross-Subject Emotion Recognition

2026-01-29

AI Summary

This work proposes an adaptive multimodal fusion framework to address performance degradation in cross-subject emotion recognition caused by modality heterogeneity and individual distribution shifts. The approach integrates multisource physiological signals through a hierarchical attention mechanism and incorporates a confidence-aware Gaussian-weighted dynamic pseudo-labeling strategy alongside a uniform alignment loss. This joint design effectively mitigates modality discrepancies, pseudo-label noise, and class imbalance while enabling robust domain adaptation. Extensive experiments on multiple cross-subject emotion recognition benchmarks demonstrate that the proposed method significantly outperforms state-of-the-art approaches in both accuracy and robustness, confirming its effectiveness and generalization capability.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Robust cross-subject emotion recognition from multimodal physiological signals remains a challenging problem, primarily due to modality heterogeneity and inter-subject distribution shift. To tackle these challenges, we propose a novel adaptive learning framework named Hierarchical Attention and Dynamic Uniform Alignment (HADUA). Our approach unifies the learning of multimodal representations with domain adaptation. First, we design a hierarchical attention module that explicitly models intra-modal temporal dynamics and inter-modal semantic interactions (e.g., between electroencephalogram(EEG) and eye movement(EM)), yielding discriminative and semantically coherent fused features. Second, to overcome the noise inherent in pseudo-labels during adaptation, we introduce a confidence-aware Gaussian weighting scheme that smooths the supervision from target-domain samples by down-weighting uncertain instances. Third, a uniform alignment loss is employed to regularize the distribution of pseudo-labels across classes, thereby mitigating imbalance and stabilizing conditional distribution matching. Extensive experiments on multiple cross-subject emotion recognition benchmarks show that HADUA consistently surpasses existing state-of-the-art methods in both accuracy and robustness, validating its effectiveness in handling modality gaps, noisy pseudo-labels, and class imbalance. Taken together, these contributions offer a practical and generalizable solution for building robust cross-subject affective computing systems.

Problem

• cross-subject emotion recognition
• multimodal physiological signals
• modality heterogeneity
• inter-subject distribution shift
• robust affective computing

Innovation

• Hierarchical Attention
• Dynamic Uniform Alignment
• Cross-Subject Emotion Recognition
• Multimodal Fusion
• Pseudo-Label Denoising