Evaluating Open-Source Vision-Language Models for Multimodal Sarcasm Detection

2025-10-13

AI Summary

This study systematically evaluates the zero-shot and few-shot capabilities of seven open-source vision-language models (VLMs)—BLIP-2, InstructBLIP, OpenFlamingo, LLaVA, PaliGemma, Gemma-3, and Qwen-VL—on multimodal sarcasm detection and explanation generation. Using three established benchmarks—Muse, MMSD2.0, and SarcNet—we conduct the first unified assessment of both image-caption sarcasm identification and visual-textual inconsistency explanation. Results show moderate performance in sarcasm classification (best F1 ≈ 62%), but consistently poor explanation quality without fine-tuning, particularly in localizing cross-modal semantic conflicts. The work identifies a critical bottleneck in current VLMs: limited interpretability in multimodal reasoning. It provides empirical evidence and a standardized evaluation framework for sarcasm understanding and trustworthy multimodal AI.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advances in open-source vision-language models (VLMs) offer new opportunities for understanding complex and subjective multimodal phenomena such as sarcasm. In this work, we evaluate seven state-of-the-art VLMs - BLIP2, InstructBLIP, OpenFlamingo, LLaVA, PaliGemma, Gemma3, and Qwen-VL - on their ability to detect multimodal sarcasm using zero-, one-, and few-shot prompting. Furthermore, we evaluate the models' capabilities in generating explanations to sarcastic instances. We evaluate the capabilities of VLMs on three benchmark sarcasm datasets (Muse, MMSD2.0, and SarcNet). Our primary objectives are twofold: (1) to quantify each model's performance in detecting sarcastic image-caption pairs, and (2) to assess their ability to generate human-quality explanations that highlight the visual-textual incongruities driving sarcasm. Our results indicate that, while current models achieve moderate success in binary sarcasm detection, they are still not able to generate high-quality explanations without task-specific finetuning.

Problem

• Evaluating VLMs for multimodal sarcasm detection
• Assessing model performance on sarcastic image-caption pairs
• Testing explanation generation for visual-textual incongruities

Innovation

• Evaluated seven open-source vision-language models for sarcasm detection
• Used zero-shot one-shot few-shot prompting techniques
• Assessed model performance on three benchmark datasets