MDPE: A Multimodal Deception Dataset with Personality and Emotional Characteristics

2024-07-17

arXiv.org

AI Summary

Existing deception detection systems are hindered by the absence of large-scale, multimodal benchmark datasets that capture individual differences—particularly personality and affective states. Method: We introduce MDPE, the first large-scale, multimodal deception dataset featuring fine-grained annotations for personality (BFI-2) and emotion (VA/CA models validated by domain experts), comprising 193 participants and over 104 hours of synchronized RGB video, audio, and transcribed text—rigorously aligned across modalities and anonymized for privacy. Contribution/Results: MDPE supports deception detection, personality recognition, affect recognition, and cross-task relational modeling. Empirical evaluation demonstrates that incorporating personality and affect features improves deception detection accuracy by up to 5.2%; notably, extraversion exhibits a significant positive correlation with deception expression intensity. MDPE fills a critical gap in interpretable, individual-difference–driven deception modeling and establishes a new benchmark for multimodal trustworthy AI.

Technology Category

• Cognitive Modeling & Cognitive Systems: Affective Computing
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Deception detection has garnered increasing attention in recent years due to the significant growth of digital media and heightened ethical and security concerns. It has been extensively studied using multimodal methods, including video, audio, and text. In addition, individual differences in deception production and detection are believed to play a crucial role.Although some studies have utilized individual information such as personality traits to enhance the performance of deception detection, current systems remain limited, partly due to a lack of sufficient datasets for evaluating performance. To address this issue, we introduce a multimodal deception dataset MDPE. Besides deception features, this dataset also includes individual differences information in personality and emotional expression characteristics. It can explore the impact of individual differences on deception behavior. It comprises over 104 hours of deception and emotional videos from 193 subjects. Furthermore, we conducted numerous experiments to provide valuable insights for future deception detection research. MDPE not only supports deception detection, but also provides conditions for tasks such as personality recognition and emotion recognition, and can even study the relationships between them. We believe that MDPE will become a valuable resource for promoting research in the field of affective computing.

Problem

• Lack of multimodal datasets for deception detection with individual differences
• Need to explore personality and emotion impacts on deceptive behavior
• Limited resources for joint deception, personality, and emotion recognition

Innovation

• Multimodal dataset with personality traits
• Includes emotional expression characteristics
• Supports deception and emotion recognition