AdS: Adapter-state Sharing Framework for Multimodal Sarcasm Detection

2025-07-06

AI Summary

In resource-constrained settings, full fine-tuning of large multimodal models for sarcasm detection is infeasible, while existing parameter-efficient fine-tuning (PEFT) methods underperform on complex cross-modal tasks. To address this, we propose AdS, a lightweight framework built upon CLIP. AdS inserts minimal adapters only into the top layers of the visual and textual encoders and introduces a novel text-guided visual adapter state-sharing mechanism to enable cross-modal collaborative learning while preserving底层 unimodal representation capabilities. By design, AdS reduces trainable parameters by over 99% compared to full fine-tuning. It achieves state-of-the-art performance on two mainstream multimodal sarcasm detection benchmarks and maintains high inference efficiency, demonstrating strong practical deployability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The growing prevalence of multimodal image-text sarcasm on social media poses challenges for opinion mining, especially under resource constraints. Existing approaches rely on full fine-tuning of large pre-trained models, making them unsuitable for low-resource settings. While recent parameter-efficient fine-tuning (PEFT) methods offer promise, their off-the-shelf use underperforms on complex tasks like sarcasm detection. We propose AdS (Adapter-State Sharing), a lightweight framework built on CLIP that inserts adapters only in the upper layers and introduces a novel adapter-state sharing mechanism, where textual adapters guide visual ones. This design promotes efficient cross-modal learning while preserving low-level unimodal representations. Experiments on two public benchmarks demonstrate that AdS achieves state-of-the-art results using significantly fewer trainable parameters than existing PEFT and full fine-tuning approaches.

Problem

• Detect multimodal sarcasm efficiently under resource constraints
• Improve parameter-efficient fine-tuning for complex sarcasm detection
• Enable cross-modal learning with lightweight adapter sharing

Innovation

• Adapter-State Sharing for cross-modal learning
• Lightweight framework built on CLIP
• Textual adapters guide visual ones