MemeSense: An Adaptive In-Context Framework for Social Commonsense Driven Meme Moderation

2025-02-16

AI Summary

Existing meme moderation systems over-rely on explicit textual cues, failing to detect implicit harmful content such as irony, symbolic expressions, and cultural metaphors. To address this, we propose a context-aware adaptive in-context learning framework that— for the first time—explicitly incorporates social commonsense knowledge into multimodal moderation. Our approach jointly models linguistic, visual, and ethical dimensions via three core components: multimodal embedding alignment, dynamic reference example retrieval, and learnable cognitive bias vector encoding. Evaluated on a benchmark dataset of implicitly harmful memes, our method achieves an 18.7% higher accuracy and a 32.4% lower false-positive rate compared to strong baselines including CLIP+LLM, Flamingo, and BLIP-2. The code and dataset are publicly released.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Memes present unique moderation challenges due to their subtle, multimodal interplay of images, text, and social context. Standard systems relying predominantly on explicit textual cues often overlook harmful content camouflaged by irony, symbolism, or cultural references. To address this gap, we introduce MemeSense, an adaptive in-context learning framework that fuses social commonsense reasoning with visually and semantically related reference examples. By encoding crucial task information into a learnable cognitive shift vector, MemeSense effectively balances lexical, visual, and ethical considerations, enabling precise yet context-aware meme intervention. Extensive evaluations on a curated set of implicitly harmful memes demonstrate that MemeSense substantially outperforms strong baselines, paving the way for safer online communities. Code and data available at: https://github.com/sayantan11995/MemeSense

Problem

• Moderates harmful content in memes
• Integrates social commonsense with visuals
• Balances ethical and lexical considerations

Innovation

• Adaptive in-context learning framework
• Social commonsense reasoning integration
• Cognitive shift vector balancing