GDCNet: Generative Discrepancy Comparison Network for Multimodal Sarcasm Detection

2026-01-28

AI Summary

This work addresses the challenge of modeling cross-modal semantic incongruity in multimodal sarcasm detection, particularly when image-text associations are weak or semantically indirect. The authors propose a novel approach that leverages multimodal large language models (MLLMs) to generate objective, fact-based image captions, which serve as stable semantic anchors to mitigate the effects of subjectivity and noise. Building upon these captions, the method computes discrepancies between the generated description and the original text in both semantic content and sentiment polarity. These discrepancies, together with image-text fidelity measures, are integrated through a gating mechanism to fuse multimodal features for sarcasm detection. Evaluated on benchmark datasets including MMSD2.0, the proposed method achieves state-of-the-art performance, significantly improving both accuracy and robustness in multimodal sarcasm detection.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multimodal sarcasm detection (MSD) aims to identify sarcasm within image-text pairs by modeling semantic incongruities across modalities. Existing methods often exploit cross-modal embedding misalignment to detect inconsistency but struggle when visual and textual content are loosely related or semantically indirect. While recent approaches leverage large language models (LLMs) to generate sarcastic cues, the inherent diversity and subjectivity of these generations often introduce noise. To address these limitations, we propose the Generative Discrepancy Comparison Network (GDCNet). This framework captures cross-modal conflicts by utilizing descriptive, factually grounded image captions generated by Multimodal LLMs (MLLMs) as stable semantic anchors. Specifically, GDCNet computes semantic and sentiment discrepancies between the generated objective description and the original text, alongside measuring visual-textual fidelity. These discrepancy features are then fused with visual and textual representations via a gated module to adaptively balance modality contributions. Extensive experiments on MSD benchmarks demonstrate GDCNet's superior accuracy and robustness, establishing a new state-of-the-art on the MMSD2.0 benchmark.

Problem

• Multimodal Sarcasm Detection
• Semantic Incongruity
• Cross-modal Misalignment
• LLM-generated Noise
• Loosely Related Modalities

Innovation

• Multimodal Sarcasm Detection
• Generative Discrepancy
• Multimodal LLMs
• Semantic Incongruity
• Gated Fusion