affective system integration

This study addresses the lack of systematic characterization of multimodal affective computing datasets for continuous valence–arousal annotation. We conduct a comprehensive survey of 25 such datasets published between 2008 and 2024, analyzing their scale, participant demographics, sensor modalities (e.g., EEG, ECG, facial video, speech), annotation protocols, and data formats. Through cross-dataset comparative analysis and methodological evaluation, we chart the technical evolution and application distribution of these resources for the first time. Our findings reveal a dominant trend toward camera-centric acquisition coupled with synergistic multimodal fusion, and quantitatively demonstrate the performance gains achievable through integrated physiological–behavioral signal fusion. The study delivers an authoritative, empirically grounded methodology guide for dataset selection, model design, and real-world deployment of affective computing systems—particularly in human–computer interaction, mental health monitoring, and autonomous driving applications.

Traditional unimodal text-based intent recognition suffers from limited contextual expressiveness, while human–computer interaction increasingly demands robust integration of heterogeneous signals. This paper systematically surveys deep learning–based multimodal intent recognition, focusing on synergistic modeling of textual, audio, visual, and physiological modalities. It traces the technical evolution from unimodal baselines to cross-modal fusion, emphasizing breakthrough applications of Transformer architectures in cross-modal alignment, feature fusion, and representation learning. We catalog 12 mainstream multimodal datasets, unify evaluation metrics, and identify representative application scenarios. A three-dimensional taxonomy—spanning modality combinations, fusion levels (early/late/hybrid), and learning paradigms (supervised/self-supervised/few-shot)—is proposed. Key challenges—including modality asynchrony, few-shot generalization, and model interpretability—are critically analyzed. Future directions include optimized cross-modal alignment, neuro-symbolic integration, and edge-efficient lightweight modeling, offering a structured reference for advancing multimodal intent understanding.

To address the challenges of deploying emotion recognition on resource-constrained edge devices—namely, stringent requirements for model compactness, ultra-low power consumption, and on-device privacy preservation—existing cloud-dependent or multimodal approaches fail to meet real-time inference demands and hardware constraints. This paper proposes an efficient microcontroller-optimized multimodal emotion recognition framework featuring audio–text late fusion. It integrates a hardware-aware quantized Transformer with a DSResNet-SE acoustic model and introduces frozen keyword embeddings for lightweight, task-specific fusion. Training-to-deployment spectral alignment is ensured via MicroFrontend preprocessing and the MLTK toolchain. Evaluated on the Coral Dev Board Micro, the system achieves end-to-end latency of 21–23 ms and memory footprint of only 1.8 MB, while improving macro-F1 by 6.3% over unimodal baselines. To our knowledge, this is the first work to enable real-time, privacy-preserving multimodal emotion inference on ultra-low-power edge hardware.

Long-term (1–2 minute) multimodal emotion recognition faces challenges in modeling dynamic cross-modal interactions and effectively fusing long video sequences with multi-channel physiological signals (e.g., EDA, ECG/PPG). To address this, we propose MVP, a lightweight attention-driven video-physiology fusion architecture. MVP introduces the first unified deep learning framework integrating a dual-stream CNN-LSTM video encoder, a time-frequency feature extraction network for physiological signals, and a cross-modal alignment module with adaptive weighted fusion. Crucially, MVP enables end-to-end co-optimization of visual and multi-channel physiological representations, substantially enhancing long-sequence modeling capability. Evaluated on standard benchmarks, MVP achieves a 4.2–6.8% absolute accuracy improvement over state-of-the-art methods under the joint video+EDA+ECG/PPG modality. Comprehensive experiments further validate its robustness and generalizability across diverse subjects and recording conditions.

This work addresses multimodal sentiment recognition and sentiment polarity analysis in realistic multi-party conversational scenarios. To overcome insufficient modeling of dynamic cross-modal coupling during multi-speaker interactions, we propose— for the first time—a four-modal (text, speech, facial, and video) collaborative modeling framework: RoBERTa, Wav2Vec 2.0, a custom lightweight FacialNet, and an end-to-end CNN-Transformer video encoder are employed for modality-specific feature extraction; features are then fused and jointly classified. Our core innovations include cross-modal temporal alignment modeling and a dialogue-oriented lightweight visual representation design. Evaluated on standard multi-party dialogue benchmarks, our method achieves 66.36% accuracy for emotion recognition and 72.15% for sentiment analysis—significantly outperforming all unimodal baselines—demonstrating the effectiveness of multimodal collaborative modeling.

This study addresses the lack of real-time responsiveness and personalization in emotional health interventions by proposing a lightweight, closed-loop affective music recommendation system. Methodologically, it integrates OpenCV-based real-time facial video capture with the lightweight DeepFace model for on-device facial emotion classification (six basic emotions), dynamically matching identified states to curated mood-regulating tracks from a locally stored, labeled music library; the system supports manual track switching, loop playback, and GUI interaction. Its key contribution is the first end-to-end, low-latency (mean 1.18 s), cloud-independent emotion–music adaptive closed loop, balancing real-time performance with user controllability. Experimental results show an emotion recognition accuracy of 89.2% and a 41% improvement in self-reported emotional well-being (N = 127), validating the efficacy and feasibility of localized music therapy for real-time emotion regulation.

This study addresses the challenge of inefficient multimodal (visual-auditory) module coordination in assistive perception systems. We propose a lightweight, domain-specific modular deep learning framework: a CNN processes eye-region images for gaze-state estimation; a deeper CNN models facial expressions (trained on FER2013); and a CNN-LSTM hybrid architecture performs speaker identification (using a custom audio dataset). Each module is independently optimized and designed for plug-and-play integration. Our key contribution is empirically validating that high-accuracy unimodal modeling combined with a loosely coupled modular architecture achieves superior performance and deployment flexibility under resource constraints. Experiments yield accuracies of 93.0% (gaze state), 97.8% (facial expression), and 96.89% (speaker ID), significantly outperforming end-to-end joint modeling baselines. This work establishes a scalable, modular paradigm for assistive technologies.

This work addresses the challenge of efficiently executing multimodal perception tasks on low-power edge devices, where existing intelligent surveillance systems struggle with both computational efficiency and context-aware resource management. We propose a real-time multimodal vision framework tailored for the Raspberry Pi 5, integrating YOLOv8n for object detection, a customized FaceNet module for face recognition, and DeepFace for emotion classification. A context-triggered adaptive runtime scheduler dynamically activates subtasks only when needed, enabling effective task coordination while substantially reducing computational load. Experimental results demonstrate a 65% reduction in computational overhead, with an object detection AP of 0.861, 88% face recognition accuracy, and an emotion classification AUC up to 0.97, achieving an overall inference speed of 5.6 FPS. These findings validate the feasibility of deploying complex multimodal AI pipelines efficiently on cost-constrained edge hardware.

This study addresses critical privacy, ethical, and cross-cultural bias challenges arising from integrating affective computing and large language models into AI systems for emotion recognition and response. Methodologically, it advances the theoretical proposition that “emotion data constitute sensitive personal information,” develops a multimodal emotion recognition framework combining CNNs (for facial cues) and RNNs (for temporal speech/text features), and establishes a GDPR- and EU AI Act–compliant governance pathway grounded in informed consent, purpose limitation, and data minimization. Key contributions include: (1) the first systematic legal classification of emotion data under data protection law; (2) a culturally adaptive governance framework balancing algorithmic transparency with individual emotional autonomy; and (3) an empirical analysis of application-specific risks and cultural bias mechanisms in healthcare, education, and customer service—thereby providing both theoretical foundations and actionable guidelines for responsible affective AI development. (149 words)

To address emotion recognition and expression challenges faced by neurodiverse individuals—particularly those with autism spectrum disorder (ASD)—this study proposes an interpretable, personalized, real-time multimodal affect estimation framework. The method integrates physiological signals (EEG, ECG, BVP, GSR/EDA) with behavioral modalities (facial expressions, speech) and models affect dynamically in the two-dimensional arousal–valence space. Key contributions include: (1) a cross-modal unified representation mechanism supporting both naturalistic (passive video viewing) and interactive (semi-structured dialogue) scenarios; and (2) an individual adaptation module coupled with neuroadaptive feedback, enabling affective education and inclusive human–machine interaction. Experimental results demonstrate significant improvements in real-time affect estimation accuracy and inter-subject specificity. This work establishes a novel paradigm for affective computing tailored to neurodiverse users, advancing both theoretical understanding and practical deployment in assistive and educational technologies.

This study addresses emotion and stress recognition from multimodal physiological signals to enhance the performance of health monitoring and affective computing systems. Leveraging the WESAD dataset, the work proposes a two-stage fusion framework that integrates early signal fusion at the sensor level with multi-model ensembling at the prediction stage. Deep temporal models—including LSTM, TCN, and Transformer—are employed to jointly model wrist- and chest-based physiological signals. This approach substantially improves system robustness and generalization, achieving state-of-the-art performance with 98.91% accuracy and a macro F1-score of 98.56% in multimodal settings.

Prior work lacks systematic evaluation of multimodal large language models (MLLMs) on fine-grained emotion understanding in open-vocabulary multimodal emotion recognition (MER-OV). Method: We introduce the first large-scale MER-OV benchmark—built upon the OV-MERD dataset—and comprehensively evaluate 19 state-of-the-art MLLMs across audio, video, and text modalities. We propose a multidimensional evaluation paradigm covering reasoning analysis, modality fusion, context utilization, and prompt engineering. Contribution/Results: Our study reveals that two-stage trimodal fusion is optimal, with video contributing most to performance; open- and closed-source MLLMs exhibit negligible performance gaps. Our framework achieves new state-of-the-art results on MER-OV. We publicly release code, models, and practical guidelines to advance interpretable, fine-grained affective AI.