I'll believe it when I see it: Images increase misinformation sharing in Vision-Language Models

2025-05-19

AI Summary

This study investigates how images influence visual-language models’ (VLMs) resharing propensity toward political news—particularly misinformation—and examines variations across model families, personality prompts, and content attributes. We propose a novel “jailbreaking-style” personalized resharing prompting strategy and construct the first multimodal political news dataset with fact-check labels and accompanying images. Leveraging PolitiFact, we establish a multimodal evaluation framework integrating Dark Triad personality and partisan-orientation prompts, conducting comparative experiments across GPT-4V, Claude-3, and LLaVA. Results show that images increase resharing of true news by 4.8% but amplify resharing of false news by 15.0%; Dark Triad personality prompts significantly exacerbate misinformation propagation risk; only Claude-3-Haiku demonstrates robustness against visual misinformation. Our work advances understanding of multimodal bias in VLMs for political discourse and provides foundational resources and benchmarks for responsible deployment.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Responsible Web: Mitigating misinformation and disinformation
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Large language models are increasingly integrated into news recommendation systems, raising concerns about their role in spreading misinformation. In humans, visual content is known to boost credibility and shareability of information, yet its effect on vision-language models (VLMs) remains unclear. We present the first study examining how images influence VLMs' propensity to reshare news content, whether this effect varies across model families, and how persona conditioning and content attributes modulate this behavior. To support this analysis, we introduce two methodological contributions: a jailbreaking-inspired prompting strategy that elicits resharing decisions from VLMs while simulating users with antisocial traits and political alignments; and a multimodal dataset of fact-checked political news from PolitiFact, paired with corresponding images and ground-truth veracity labels. Experiments across model families reveal that image presence increases resharing rates by 4.8% for true news and 15.0% for false news. Persona conditioning further modulates this effect: Dark Triad traits amplify resharing of false news, whereas Republican-aligned profiles exhibit reduced veracity sensitivity. Of all the tested models, only Claude-3-Haiku demonstrates robustness to visual misinformation. These findings highlight emerging risks in multimodal model behavior and motivate the development of tailored evaluation frameworks and mitigation strategies for personalized AI systems. Code and dataset are available at: https://github.com/3lis/misinfo_vlm

Problem

• Images increase misinformation sharing in Vision-Language Models
• Effect of images on VLMs' news resharing behavior unclear
• Persona conditioning and content attributes influence misinformation spread

Innovation

• Jailbreaking-inspired prompting for VLM resharing decisions
• Multimodal dataset with fact-checked news and images
• Persona conditioning to study misinformation resharing effects