Faces speak louder than words: Emotions versus textual sentiment in the 2024 USA Presidential Election

2024-12-23

arXiv.org

AI Summary

This study investigates the coordination mechanism and temporal evolution of facial expression sentiment and textual sentiment in political images on Instagram during the 2024 U.S. presidential election. Using multimodal data from April 5 to August 9, 2024, we integrate FaceNet for facial feature extraction, AffectNet for fine-grained facial emotion classification, and BERT-based textual sentiment analysis, augmented by event-driven temporal comparative modeling. Results reveal strong overall alignment (r = 0.73) yet critical divergence: facial expressions exhibit 12% higher sensitivity to pivotal political events and convey polarity-transcendent political semantics. Crucially, we identify for the first time party-specific strategic deployment of distinct facial emotions—e.g., fear (Democrats) versus anger (Republicans)—to construct antagonistic discourses; moreover, neutral and negative facial expressions serve significant rhetorical functions. These findings advance multimodal political communication research by establishing a novel affective analytics framework that jointly models visual and linguistic sentiment dynamics across time and events.

Technology Category

• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Sentiment analysis of textual content has become a well-established solution for analyzing social media data. However, with the rise of images and videos as primary modes of expression, more information on social media is conveyed visually. Among these, facial expressions serve as one of the most direct indicators of emotional content in images. This study analyzes a dataset of Instagram posts related to the 2024 U.S. presidential election, spanning April 5, 2024, to August 9, 2024, to compare the relationship between textual and facial sentiment. Our findings reveal that facial expressions generally align with text sentiment, although neutral and negative facial expressions provide critical information beyond valence. Furthermore, during politically significant events such as Donald Trump's conviction and assassination attempt, posts depicting Trump showed a 12% increase in negative sentiment. Crucially, Democrats use their opponent's fear to depict weakness whereas Republicans use their candidate's anger to depict resilience. Our research highlights the potential of integrating facial expression analysis with textual sentiment analysis to uncover deeper insights into social media dynamics.

Problem

• Compare facial and textual sentiment in election posts
• Analyze emotional impact of political events on sentiment
• Integrate facial and text analysis for deeper insights

Innovation

• Analyzing facial expressions for emotional sentiment
• Comparing textual and facial sentiment alignment
• Integrating facial and text analysis for insights