Deep Learning for Climate Action: Computer Vision Analysis of Visual Narratives on X

2025-03-12

AI Summary

This study addresses the challenge of climate-related visual narrative analysis on social media in the post-API era. We propose the first systematic multimodal framework to analyze 730,000 climate-related tweets and associated images from X (formerly Twitter) in 2019. Methodologically, we integrate ResNet/ViT for image classification, YOLOv8 for object detection, VADER/FinBERT for textual sentiment analysis, CLIP for cross-modal alignment, and Shapley values for explainable visualization, implemented via a Streamlit-based interactive GUI toolchain. Key contributions include: (1) the first identification of six dominant climate visual themes; (2) a novel quantitative framework for measuring image–text sentiment divergence, revealing significant inconsistency in 32% of samples; (3) empirical validation that CLIP’s accuracy drops to 61.4% on real-world social media imagery, underscoring the necessity of domain adaptation; and (4) open-sourcing a fully reproducible toolchain with annotation interfaces to advance interpretable, multimodal climate communication research.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

Climate change is one of the most pressing challenges of the 21st century, sparking widespread discourse across social media platforms. Activists, policymakers, and researchers seek to understand public sentiment and narratives while access to social media data has become increasingly restricted in the post-API era. In this study, we analyze a dataset of climate change-related tweets from X (formerly Twitter) shared in 2019, containing 730k tweets along with the shared images. Our approach integrates statistical analysis, image classification, object detection, and sentiment analysis to explore visual narratives in climate discourse. Additionally, we introduce a graphical user interface (GUI) to facilitate interactive data exploration. Our findings reveal key themes in climate communication, highlight sentiment divergence between images and text, and underscore the strengths and limitations of foundation models in analyzing social media imagery. By releasing our code and tools, we aim to support future research on the intersection of climate change, social media, and computer vision.

Problem

• Analyzing public sentiment on climate change via social media imagery.
• Exploring visual narratives in climate discourse using computer vision.
• Developing tools for interactive data exploration in climate communication.

Innovation

• Integrates image classification and object detection
• Utilizes sentiment analysis for climate discourse
• Develops GUI for interactive data exploration