Dynamic Emotion and Personality Profiling for Multimodal Deception Detection

2026-04-18

AI Summary

Existing approaches to multimodal deception detection are hindered by the absence of sample-level dynamic annotations for emotion and personality, limiting their effectiveness. To address this gap, this work introduces a multi-model, multi-prompt annotation protocol alongside rigorous label quality evaluation criteria, resulting in DDEP—the first multimodal dataset jointly annotated for deception, emotion, and personality. Furthermore, the authors propose Rel-DDEP, an adaptive reliability-weighted fusion framework that models uncertainty via high-dimensional Gaussian distributions and incorporates alignment and ranking constraints to enable joint learning across the three tasks. Experiments on both MDPE and the newly curated DDEP demonstrate that the proposed method outperforms current state-of-the-art models, achieving absolute F1-score improvements of 2.53%, 2.66%, and 9.30% in deception, emotion, and personality detection, respectively.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Deception detection is of great significance for ensuring information security and conducting public opinion analysis, with personality factors and emotion cues playing a critical role. However, existing methods lack sample-level dynamic annotations for emotions and personality.In this paper, we propose an innovative multi-model multi-prompt annotation scheme and a strict label quality evaluation standard, and establish a multimodal joint detection dataset DDEP for deception, emotion, and personality. Meanwhile, we propose Rel-DDEP, an adaptive reliability-weighted fusion framework. Our framework quantifies uncertainty by mapping modal features to a high-dimensional Gaussian distribution space. It then performs reliability-weighted fusion and incorporates an alignment module and a sorting constraint module to achieve joint detection of deception, emotion, and personality. Experimental results on the MDPE and DDEP datasets show that our Rel-DDEP significantly outperforms the existing state-of-the-art baseline models in three tasks. The F1 score of the deception detection increases by 2.53%, that of the emotion detection increases by 2.66%, and that of the personality detection increases by 9.30%. The experiments fully verify the necessity of annotating dynamic emotion and personality labels for each sample and the effectiveness of reliability-weighted fusion.

Problem

• deception detection
• dynamic emotion
• personality profiling
• multimodal
• sample-level annotation

Innovation

• dynamic annotation
• reliability-weighted fusion
• multimodal deception detection
• personality profiling
• uncertainty quantification