VS-Bench: Evaluating VLMs for Strategic Reasoning and Decision-Making in Multi-Agent Environments

2025-06-03

AI Summary

Existing VLM evaluation benchmarks are limited to single-agent or purely textual settings, failing to assess strategic multi-agent interactions in vision-language grounded environments. To address this gap, we introduce VS-Bench—the first benchmark for evaluating multi-agent vision-language strategic reasoning—covering eight embodied, motivationally heterogeneous scenarios (cooperative, competitive, and mixed). Methodologically, we propose a dual-dimensional evaluation framework: offline (action prediction accuracy) and online (normalized episode return), implemented within a multi-agent reinforcement learning environment that supports vision-grounded tasks, theory-of-mind modeling, and social behavior analysis. Experiments across 14 state-of-the-art VLMs reveal severe limitations: the best-performing model achieves only 47.8% action prediction accuracy and 24.3% normalized episode return, underscoring fundamental deficits in long-horizon planning and multi-agent modeling. VS-Bench thus establishes a foundational benchmark for advancing strategic, multimodal, multi-agent reasoning in VLMs.

Technology Category

• Multiagent Systems: Adversarial Agents
• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent advancements in Vision Language Models (VLMs) have expanded their capabilities to interactive agent tasks, yet existing benchmarks remain limited to single-agent or text-only environments. In contrast, real-world scenarios often involve multiple agents interacting within rich visual and linguistic contexts, posing challenges with both multimodal observations and strategic interactions. To bridge this gap, we introduce Visual Strategic Bench (VS-Bench), a multimodal benchmark that evaluates VLMs for strategic reasoning and decision-making in multi-agent environments. VS-Bench comprises eight vision-grounded environments spanning cooperative, competitive, and mixed-motive interactions, designed to assess agents' ability to predict others' future moves and optimize for long-term objectives. We consider two complementary evaluation dimensions, including offline evaluation of strategic reasoning by next-action prediction accuracy and online evaluation of decision-making by normalized episode return. Extensive experiments of fourteen leading VLMs reveal a significant gap between current models and optimal performance, with the best models attaining 47.8% prediction accuracy and 24.3% normalized return. We further conduct in-depth analyses on multimodal observations, test-time scaling, social behaviors, and failure cases of VLM agents. By standardizing the evaluation and highlighting the limitations of existing models, we envision VS-Bench as a foundation for future research on strategic multimodal agents. Code and data are available at https://vs-bench.github.io.

Problem

• Evaluating VLMs in multi-agent strategic reasoning
• Assessing multimodal decision-making in interactive environments
• Bridging gap between single-agent and real-world scenarios

Innovation

• Multimodal benchmark for multi-agent strategic reasoning
• Evaluates VLMs with vision-grounded cooperative and competitive tasks
• Measures prediction accuracy and normalized return performance