🤖 AI Summary
This study investigates how visual inputs influence the behavior of vision-language models (VLMs) in cooperative decision-making, with a particular focus on the risk of uncontrolled actions in safety-critical scenarios. Using an iterated prisoner’s dilemma framework, the authors systematically evaluate the impact of semantic images and color-coded reward matrices on VLMs’ propensity to cooperate, while also assessing mitigation strategies such as prompt engineering, chain-of-thought reasoning, and visual token reduction. The work reveals for the first time that visual priming significantly alters VLM cooperation behavior and demonstrates substantial differences in susceptibility across model architectures. These findings underscore the necessity of architecture-specific robustness evaluations prior to real-world deployment.
📝 Abstract
As Vision-Language Models (VLMs) become increasingly integrated into decision-making systems, it is essential to understand how visual inputs influence their behavior. This paper investigates the effects of visual priming on VLMs' cooperative behavior using the Iterated Prisoner's Dilemma (IPD) as a test scenario. We examine whether exposure to images depicting behavioral concepts (kindness/helpfulness vs. aggressiveness/selfishness) and color-coded reward matrices alters VLM decision patterns. Experiments were conducted across multiple state-of-the-art VLMs. We further explore mitigation strategies including prompt modifications, Chain of Thought (CoT) reasoning, and visual token reduction. Results show that VLM behavior can be influenced by both image content and color cues, with varying susceptibility and mitigation effectiveness across models. These findings not only underscore the importance of robust evaluation frameworks for VLM deployment in visually rich and safety-critical environments, but also highlight how architectural and training differences among models may lead to distinct behavioral responses-an area worthy of further investigation.