MUStReason: A Benchmark for Diagnosing Pragmatic Reasoning in Video-LMs for Multimodal Sarcasm Detection

2025-10-27

AI Summary

Multimodal sarcasm detection requires cross-modal cue integration and pragmatic inference, yet existing video-language models (VideoLMs) exhibit significant limitations in interpreting such implicit intentions. Method: We introduce MUStReason, a diagnostic benchmark featuring systematic annotations of modality-specific cues and fine-grained reasoning steps, exposing weaknesses in cross-modal perception and intent inference. We further propose PragCoT—a chain-of-thought prompting framework that decouples sarcasm detection into explicit “perception” and “pragmatic reasoning” stages, guiding models to transcend literal semantics and model non-literal intent. Contribution/Results: Through quantitative evaluation and qualitative attribution analysis, we achieve interpretable assessment of reasoning processes. Experiments demonstrate that PragCoT substantially enhances VideoLMs’ pragmatic understanding, establishing a new benchmark and methodological paradigm for multimodal pragmatic reasoning research.

Technology Category

• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Sarcasm is a specific type of irony which involves discerning what is said from what is meant. Detecting sarcasm depends not only on the literal content of an utterance but also on non-verbal cues such as speaker's tonality, facial expressions and conversational context. However, current multimodal models struggle with complex tasks like sarcasm detection, which require identifying relevant cues across modalities and pragmatically reasoning over them to infer the speaker's intention. To explore these limitations in VideoLMs, we introduce MUStReason, a diagnostic benchmark enriched with annotations of modality-specific relevant cues and underlying reasoning steps to identify sarcastic intent. In addition to benchmarking sarcasm classification performance in VideoLMs, using MUStReason we quantitatively and qualitatively evaluate the generated reasoning by disentangling the problem into perception and reasoning, we propose PragCoT, a framework that steers VideoLMs to focus on implied intentions over literal meaning, a property core to detecting sarcasm.

Problem

• Diagnosing pragmatic reasoning limitations in multimodal sarcasm detection
• Identifying relevant cross-modal cues for sarcasm interpretation in VideoLMs
• Improving VideoLMs' ability to infer implied intentions beyond literal meaning

Innovation

• Proposes PragCoT framework for implied intention focus
• Disentangles problem into perception and reasoning steps
• Enriches benchmark with modality-specific cue annotations