Hi-EF: Benchmarking Emotion Forecasting in Human-interaction

2024-07-23

arXiv.org

AI Summary

Traditional emotion prediction methods rely heavily on exogenous variables and struggle to model the dynamic emotional evolution inherent in dyadic interactions, often remaining at a qualitative level. This paper proposes Interaction-Driven Emotion Forecasting (EF), a novel paradigm that reframes psychological affective forecasting as a multimodal deep learning problem, integrating emotion contagion and context-dependency mechanisms to model individuals’ future emotional states. Our contributions are threefold: (1) We formally define the EF task; (2) We introduce Hi-EF—the first benchmark dataset for dyadic emotion forecasting with hierarchical contextual annotations (3,069 samples, covering text, audio, and visual modalities); (3) We design a unified framework comprising context-aware sequential modeling, interactive emotion propagation modeling, and hierarchical supervised learning. Extensive experiments validate the feasibility of EF: baseline models achieve significant improvements over conventional approaches in cross-modal emotion trend prediction. Both the Hi-EF dataset and source code are publicly released.

Technology Category

• Cognitive Modeling & Cognitive Systems: Affective Computing
• Machine Learning: Multimodal Learning
• Humans and AI: Emotional Intelligence

Application Category

• User Modeling, Personalization and Recommendation: Psychology-informed user models and recommender systems
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Affective Forecasting, a research direction in psychology that predicts individuals future emotions, is often constrained by numerous external factors like social influence and temporal distance. To address this, we transform Affective Forecasting into a Deep Learning problem by designing an Emotion Forecasting paradigm based on two-party interactions. We propose a novel Emotion Forecasting (EF) task grounded in the theory that an individuals emotions are easily influenced by the emotions or other information conveyed during interactions with another person. To tackle this task, we have developed a specialized dataset, Human-interaction-based Emotion Forecasting (Hi-EF), which contains 3069 two-party Multilayered-Contextual Interaction Samples (MCIS) with abundant affective-relevant labels and three modalities. Hi-EF not only demonstrates the feasibility of the EF task but also highlights its potential. Additionally, we propose a methodology that establishes a foundational and referential baseline model for the EF task and extensive experiments are provided. The dataset and code is available at https://github.com/Anonymize-Author/Hi-EF.

Problem

• Benchmarks emotion forecasting in human interactions
• Predicts emotional responses using partner cues
• Enhances feasibility of structured emotion forecasting

Innovation

• Introduces Human-interaction-based Emotion Forecasting task
• Focuses on two-party interaction for structured emotional analysis
• Uses partner's emotional cues to predict individual's future emotions