Teaching Sarcasm: Few-Shot Multimodal Sarcasm Detection via Distillation to a Parameter-Efficient Student

2025-10-29

AI Summary

In low-resource multimodal sarcasm detection, modeling image-text incongruence remains challenging, and limited supervision in few-shot settings constrains the effectiveness of parameter-efficient fine-tuning (PEFT) methods. To address these issues, this paper proposes PEKD, an entropy-aware knowledge distillation framework tailored for PEFT. Its core contributions are twofold: (1) an entropy-aware gating mechanism that dynamically suppresses high-uncertainty distillation signals from the teacher model, thereby improving distillation reliability; and (2) a modular distillation architecture that seamlessly integrates with diverse PEFT student models—including LoRA, Adapters, and Prompt Tuning—without architectural modification. Extensive few-shot experiments on two benchmark multimodal datasets demonstrate that PEKD consistently outperforms existing PEFT approaches and large multimodal baselines, achieving state-of-the-art performance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Multimodal sarcasm detection is challenging, especially in low-resource settings where subtle image-text contradictions are hard to learn due to scarce annotated data, which hinders the model's performance. Parameter-efficient fine-tuning (PEFT) methods like adapters, LoRA, and prompt tuning reduce overfitting but struggle to reach optimal performance due to limited supervision from few-shot data. We propose PEKD, a unified framework that enhances PEFT methods via distillation from an expert model trained on large-scale sarcasm data, which acts as the teacher. To mitigate unreliable signals from the teacher, we introduce an entropy-aware gating mechanism that dynamically adjusts the distillation strength based on teacher confidence. Experiments on two public datasets demonstrate that our PEKD framework enables PEFT methods to outperform both prior parameter-efficient approaches and large multimodal models, achieving strong results in the few-shot scenario. The framework is modular and adaptable to a wide range of multimodal models and tasks.

Problem

• Detecting sarcasm from limited multimodal data
• Improving parameter-efficient methods with few-shot supervision
• Mitigating unreliable signals in multimodal sarcasm detection

Innovation

• Distillation from expert model enhances PEFT methods
• Entropy-aware gating adjusts distillation strength dynamically
• Modular framework adapts to multimodal models and tasks