Improving Multimodal Hateful Meme Detection Exploiting LMM-Generated Knowledge

2025-04-14

AI Summary

This work addresses the challenge of identity-based multimodal hate meme detection in social media. We propose a large multimodal model (LMM)-driven, knowledge-enhanced approach. Methodologically, we introduce a novel dual-path LMM knowledge utilization paradigm: (1) task-oriented representation extraction to generate fine-grained semantic and affective descriptions; and (2) a knowledge-injection-based hard example mining mechanism that explicitly integrates LMM-implicit semantic and affective priors. Technically, our framework unifies CLIP-style multimodal encoding, knowledge distillation, and fine-tuned classification heads. Our method achieves state-of-the-art performance on two mainstream benchmarks. The code and models are publicly released. The core contribution lies in the first systematic integration of deep semantic and affective knowledge from LMMs into the entire multimodal hate detection pipeline, significantly improving detection accuracy—particularly for subtle, context-dependent hate memes.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Memes have become a dominant form of communication in social media in recent years. Memes are typically humorous and harmless, however there are also memes that promote hate speech, being in this way harmful to individuals and groups based on their identity. Therefore, detecting hateful content in memes has emerged as a task of critical importance. The need for understanding the complex interactions of images and their embedded text renders the hateful meme detection a challenging multimodal task. In this paper we propose to address the aforementioned task leveraging knowledge encoded in powerful Large Multimodal Models (LMM). Specifically, we propose to exploit LMMs in a two-fold manner. First, by extracting knowledge oriented to the hateful meme detection task in order to build strong meme representations. Specifically, generic semantic descriptions and emotions that the images along with their embedded texts elicit are extracted, which are then used to train a simple classification head for hateful meme detection. Second, by developing a novel hard mining approach introducing directly LMM-encoded knowledge to the training process, providing further improvements. We perform extensive experiments on two datasets that validate the effectiveness of the proposed method, achieving state-of-the-art performance. Our code and trained models are publicly available at: https://github.com/IDT-ITI/LMM-CLIP-meme.

Problem

• Detect hate speech in multimodal memes effectively
• Leverage Large Multimodal Models for enhanced detection
• Improve classification with LMM-generated knowledge and hard mining

Innovation

• Leveraging Large Multimodal Models for knowledge extraction
• Building strong meme representations using semantic descriptions
• Introducing LMM-encoded knowledge via hard mining approach