From Shallow Humor to Metaphor: Towards Label-Free Harmful Meme Detection via LMM Agent Self-Improvement

2025-12-25

AI Summary

The proliferation of harmful memes online poses significant challenges, exacerbated by the limitations of existing detection methods—which rely heavily on large-scale annotated data and struggle to adapt to rapidly evolving meme content. Method: This paper introduces ALARM, the first annotation-free detection framework, leveraging large multimodal models (LMMs) to establish a self-improving proxy mechanism: (1) confidence-based identification of superficial, explicit harmful memes to generate high-quality pseudo-labels; and (2) pairwise contrastive learning to progressively train the LMM on complex implicit harmful patterns—such as metaphor and irony. Contribution/Results: Evaluated on three heterogeneous datasets, ALARM substantially outperforms supervised baselines and demonstrates strong generalization and rapid adaptation to novel, evolved memes. These results validate the effectiveness and scalability of the annotation-free paradigm in dynamic online environments.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The proliferation of harmful memes on online media poses significant risks to public health and stability. Existing detection methods heavily rely on large-scale labeled data for training, which necessitates substantial manual annotation efforts and limits their adaptability to the continually evolving nature of harmful content. To address these challenges, we present ALARM, the first lAbeL-free hARmful Meme detection framework powered by Large Multimodal Model (LMM) agent self-improvement. The core innovation of ALARM lies in exploiting the expressive information from "shallow" memes to iteratively enhance its ability to tackle more complex and subtle ones. ALARM consists of a novel Confidence-based Explicit Meme Identification mechanism that isolates the explicit memes from the original dataset and assigns them pseudo-labels. Besides, a new Pairwise Learning Guided Agent Self-Improvement paradigm is introduced, where the explicit memes are reorganized into contrastive pairs (positive vs. negative) to refine a learner LMM agent. This agent autonomously derives high-level detection cues from these pairs, which in turn empower the agent itself to handle complex and challenging memes effectively. Experiments on three diverse datasets demonstrate the superior performance and strong adaptability of ALARM to newly evolved memes. Notably, our method even outperforms label-driven methods. These results highlight the potential of label-free frameworks as a scalable and promising solution for adapting to novel forms and topics of harmful memes in dynamic online environments.

Problem

• Detects harmful memes without labeled data
• Uses LMM self-improvement to handle complex memes
• Adapts to evolving harmful content online

Innovation

• Label-free harmful meme detection using LMM agent self-improvement
• Confidence-based explicit meme identification with pseudo-labeling
• Pairwise learning guided agent self-improvement with contrastive pairs