Adaptive Physical-Facial Representation Fusion via Subject-Invariant Cross-Modal Prompt Tuning for Video-Based Emotion Recognition

2026-05-07

AI Summary

This work addresses the limitations of existing video-based emotion recognition methods, which often degrade pretrained facial representations when fusing facial and remote photoplethysmography (rPPG) signals and suffer from poor cross-subject generalization due to individual variability. To overcome these issues, the authors propose a subject-invariant cross-modal prompt tuning mechanism that converts rPPG signals into noise-robust time-frequency representations and uses them as prompts to modulate facial features within a frozen Vision Transformer. Additionally, a Decoupled Shared-Specific Adapter (DSSA) is introduced to disentangle subject-shared and subject-specific components. This approach preserves the integrity of facial representations while significantly enhancing generalization performance, achieving state-of-the-art accuracy and superior cross-subject robustness on both the MAHNOB-HCI and DEAP datasets compared to strong baselines.

Abstract

Emotion recognition from facial videos enables non-contact inference of human emotional states. Although facial expressions are widely used cues, they cannot fully reflect intrinsic affective states. Remote photoplethysmography (rPPG) provides complementary physiological information, but it is highly susceptible to noise and inter-subject variability, limiting generalization to unseen individuals. Existing multimodal methods combine facial and rPPG features, yet their fusion strategies often disrupt pretrained facial representations and lack explicit mechanisms to suppress subject-specific variations. To address these issues, we propose a subject-invariant cross-modal prompt-tuning framework for video-based emotion recognition. Specifically, rPPG waveforms are transformed into noise-robust time-frequency representations (TFRs), from which modality-complementary prompts are generated to modulate facial tokens within a frozen Vision Transformer (ViT). This design enables effective cross-modal interaction while preserving the generalizable facial representations learned by the pretrained backbone. In addition, we introduce a decoupled shared-specific adapter (DSSA) into each ViT layer to explicitly separate subject-shared and subject-specific components, thereby improving cross-subject generalization. Experiments on the MAHNOB-HCI and DEAP benchmarks demonstrate that the proposed method consistently outperforms strong baselines in both recognition accuracy and generalization ability, highlighting its effectiveness for video-based emotion recognition.

Problem

• emotion recognition
• remote photoplethysmography
• cross-modal fusion
• subject-invariant
• video-based

Innovation

• cross-modal prompt tuning
• subject-invariant representation
• remote photoplethysmography (rPPG)
• Vision Transformer (ViT)
• decoupled shared-specific adapter