Dual Modality-Aware Gated Prompt Tuning for Few-Shot Multimodal Sarcasm Detection

2025-07-06

AI Summary

To address performance degradation in few-shot multimodal sarcasm detection caused by scarce annotated data, this paper proposes GPTuner, a bimodal-aware gated prompt tuning framework. GPTuner injects modality-specific deep prompts hierarchically and employs a gating mechanism to enable fine-grained interaction between textual and visual features. It further introduces cross-layer prompt sharing and cross-modal prompt alignment modules to enhance high-level semantic fusion while promoting modality disentanglement. Compared with conventional fine-tuning and existing prompt-based methods, GPTuner significantly improves sarcasm recognition under low-resource conditions. On two public benchmarks, it achieves average accuracy gains of 4.2–7.8 percentage points under 1-shot and 5-shot settings. Moreover, GPTuner demonstrates strong cross-domain generalization capability, validating its robustness and adaptability across diverse data distributions.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

The widespread use of multimodal content on social media has heightened the need for effective sarcasm detection to improve opinion mining. However, existing models rely heavily on large annotated datasets, making them less suitable for real-world scenarios where labeled data is scarce. This motivates the need to explore the problem in a few-shot setting. To this end, we introduce DMDP (Deep Modality-Disentangled Prompt Tuning), a novel framework for few-shot multimodal sarcasm detection. Unlike prior methods that use shallow, unified prompts across modalities, DMDP employs gated, modality-specific deep prompts for text and visual encoders. These prompts are injected across multiple layers to enable hierarchical feature learning and better capture diverse sarcasm types. To enhance intra-modal learning, we incorporate a prompt-sharing mechanism across layers, allowing the model to aggregate both low-level and high-level semantic cues. Additionally, a cross-modal prompt alignment module enables nuanced interactions between image and text representations, improving the model's ability to detect subtle sarcastic intent. Experiments on two public datasets demonstrate DMDP's superior performance in both few-shot and extremely low-resource settings. Further cross-dataset evaluations show that DMDP generalizes well across domains, consistently outperforming baseline methods.

Problem

• Detecting sarcasm in multimodal social media content with limited labeled data
• Improving few-shot learning for text and visual sarcasm detection
• Enhancing cross-modal alignment for subtle sarcasm intent recognition

Innovation

• Modality-specific deep prompts for text and visual encoders
• Prompt-sharing mechanism across layers for intra-modal learning
• Cross-modal prompt alignment for nuanced interactions