Probing Multimodal Large Language Models on Cognitive Biases in Chinese Short-Video Misinformation

2026-01-10

arXiv.org

AI Summary

This study systematically evaluates the robustness of multimodal large language models (MLLMs) in detecting false health information in Chinese short videos, revealing for the first time their vulnerability to cognitive biases—particularly susceptibility to cues from authoritative accounts. We construct a high-quality dataset of 200 expert-annotated samples encompassing three representative deception patterns and introduce an evidence verification framework grounded in national standards and academic literature, along with a fine-grained belief scoring mechanism. Evaluations across five modality configurations on eight state-of-the-art MLLMs show that Gemini-2.5-Pro achieves the highest performance (belief score: 71.5/100), while o3 performs weakest (35.2), underscoring current models’ overreliance on social cues rather than substantive evidence.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Short-video platforms have become major channels for misinformation, where deceptive claims frequently leverage visual experiments and social cues. While Multimodal Large Language Models (MLLMs) have demonstrated impressive reasoning capabilities, their robustness against misinformation entangled with cognitive biases remains under-explored. In this paper, we introduce a comprehensive evaluation framework using a high-quality, manually annotated dataset of 200 short videos spanning four health domains. This dataset provides fine-grained annotations for three deceptive patterns, experimental errors, logical fallacies, and fabricated claims, each verified by evidence such as national standards and academic literature. We evaluate eight frontier MLLMs across five modality settings. Experimental results demonstrate that Gemini-2.5-Pro achieves the highest performance in the multimodal setting with a belief score of 71.5/100, while o3 performs the worst at 35.2. Furthermore, we investigate social cues that induce false beliefs in videos and find that models are susceptible to biases like authoritative channel IDs.

Problem

• Multimodal Large Language Models
• Cognitive Biases
• Short-Video Misinformation
• Deceptive Patterns
• Social Cues

Innovation

• Multimodal Large Language Models
• Cognitive Biases
• Short-Video Misinformation
• Evaluation Framework
• Deceptive Patterns