MemeIntel: Explainable Detection of Propagandistic and Hateful Memes

2025-02-23

AI Summary

This work addresses the automatic detection and explainable prediction of Arabic propaganda and English hate memes. We introduce the first large-scale bilingual (Arabic/English) explainable meme dataset, featuring fine-grained labels and human-readable rationale generation. Methodologically, we propose the first multi-stage optimization-driven joint training framework for vision-language models (VLMs), integrating cross-modal alignment with explanation-guided fine-tuning to simultaneously enhance detection accuracy and explanation quality. On the ArMeme and Hateful Memes benchmarks, our approach achieves absolute improvements of 3.0% and 7.2% in detection accuracy, respectively, outperforming state-of-the-art methods. Our core contributions are threefold: (1) the first bilingual explainable meme benchmark; (2) a novel VLM training paradigm that jointly optimizes detection and explanation; and (3) advancement of trustworthy AI for multimodal content safety.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The proliferation of multimodal content on social media presents significant challenges in understanding and moderating complex, context-dependent issues such as misinformation, hate speech, and propaganda. While efforts have been made to develop resources and propose new methods for automatic detection, limited attention has been given to label detection and the generation of explanation-based rationales for predicted labels. To address this challenge, we introduce MemeIntel, an explanation-enhanced dataset for propaganda memes in Arabic and hateful memes in English, making it the first large-scale resource for these tasks. To solve these tasks, we propose a multi-stage optimization approach and train Vision-Language Models (VLMs). Our results demonstrate that this approach significantly improves performance over the base model for both extbf{label detection} and explanation generation, outperforming the current state-of-the-art with an absolute improvement of ~3% on ArMeme and ~7% on Hateful Memes. For reproducibility and future research, we aim to make the MemeIntel dataset and experimental resources publicly available.

Problem

• Detect propagandistic and hateful memes
• Generate explanation-based rationales
• Improve label detection performance

Innovation

• Explanation-enhanced dataset creation
• Multi-stage optimization approach
• Vision-Language Models training