Enhancing multimodal affect recognition in healthcare: the robustness of appraisal dimensions over labels within age groups and in cross-age generalisation

2026-04-30

AI Summary

This study addresses the challenge of limited cross-age generalization in affective computing for healthcare AI. To this end, the authors construct a multimodal emotional dataset encompassing both older and younger adults and present the first systematic comparison between dimensional (appraisal-based) and categorical emotion recognition models across within-group and cross-age scenarios. Leveraging multimodal fusion, deep representation learning, and temporal continuity modeling, the findings demonstrate that dimensional models consistently outperform categorical approaches under all evaluation conditions. Notably, dimensional models retain predictive performance significantly above chance even in cross-corpus testing, underscoring their superior robustness and generalizability. The project further releases an open-source API to facilitate emotion measurement applications in behavioral science.

Abstract

The integration of artificial intelligence (AI) into healthcare has advanced significantly, yet affect recognition remains a major challenge, particularly in AI-assisted interventions such as Computerized Cognitive Training (CCT). The THERADIA-WoZ corpus was developed to enable multimodal affect recognition in the context of AI-driven CCT, focusing on an older adult population. This study extends the corpus by introducing a dataset collected from young adults, allowing direct comparison of affect recognition models across age groups. Our objective was to assess whether multimodal models based on dimensions borrowed from appraisal theories outperform those based on categorical labels and to evaluate their generalisation power across age corpora. After comparing both corpora, models were trained and tested using within-corpus, cross-corpus, and mixed-corpus evaluation. Results revealed that appraisal dimensions consistently outperformed categorical labels across all conditions, demonstrating greater predictive accuracy and stability. Notably, categorical labels failed to generalise across age corpora, as performance dropped to chance levels in cross-corpus evaluation. In contrast, appraisal dimensions maintained predictive performance above chance, reinforcing their robustness for cross-age affect recognition. Furthermore, training on both corpora did not improve generalisation beyond within-corpus training. The findings support the theoretical and practical advantages of appraisal dimensions over categorical labels in affective computing. They also highlight the importance of multimodal fusion and deep learning representations for emotion modeling. To facilitate future research, we provide an API for researchers interested in time-continuous emotion prediction, offering valuable tools for behavioral sciences to enhance the measurement of emotional states in various experimental settings.

Problem

• affect recognition
• cross-age generalisation
• appraisal dimensions
• multimodal emotion modeling
• healthcare AI

Innovation

• appraisal dimensions
• multimodal affect recognition
• cross-age generalisation
• emotional modeling
• deep learning representations