Think Twice Before You Judge: Mixture of Dual Reasoning Experts for Multimodal Sarcasm Detection

2025-07-06

AI Summary

Existing methods predominantly rely on shallow multimodal cues, failing to capture the deep cultural commonsense knowledge and logical reasoning essential for irony detection. To address this, we propose a Dual-Reasoning Mixture-of-Experts framework: (1) fine-grained intra-modal alignment to model internal image–text inconsistency, and (2) large vision-language model (VLM)-guided chain-of-thought external reasoning to generate structured commonsense justifications. An adaptive gating mechanism dynamically fuses these two complementary reasoning pathways. Our approach achieves significant improvements over state-of-the-art methods on two mainstream multimodal irony detection benchmarks. Ablation studies confirm that, despite inherent noise in external reasoning, it substantially enhances the model’s capacity to infer implicit ironic intent. To our knowledge, this is the first work to jointly leverage internal inconsistency modeling and VLM-augmented, structured external reasoning for irony detection—establishing a novel paradigm for deep multimodal semantic understanding.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multimodal sarcasm detection has attracted growing interest due to the rise of multimedia posts on social media. Understanding sarcastic image-text posts often requires external contextual knowledge, such as cultural references or commonsense reasoning. However, existing models struggle to capture the deeper rationale behind sarcasm, relying mainly on shallow cues like image captions or object-attribute pairs from images. To address this, we propose extbf{MiDRE} ( extbf{Mi}xture of extbf{D}ual extbf{R}easoning extbf{E}xperts), which integrates an internal reasoning expert for detecting incongruities within the image-text pair and an external reasoning expert that utilizes structured rationales generated via Chain-of-Thought prompting to a Large Vision-Language Model. An adaptive gating mechanism dynamically weighs the two experts, selecting the most relevant reasoning path. Experiments on two benchmark datasets show that MiDRE achieves superior performance over baselines. Various qualitative analyses highlight the crucial role of external rationales, revealing that even when they are occasionally noisy, they provide valuable cues that guide the model toward a better understanding of sarcasm.

Problem

• Detecting sarcasm in multimodal image-text posts
• Addressing reliance on shallow cues in sarcasm detection
• Integrating internal and external reasoning for sarcasm understanding

Innovation

• Mixture of dual reasoning experts for sarcasm detection
• Internal and external reasoning experts integration
• Adaptive gating mechanism for dynamic path selection