RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Models

2026-06-17

AI Summary

This work addresses the limited strategic reasoning capabilities of current vision-language models in complex adversarial and cooperative scenarios, as well as the shortcomings of existing real-time strategy (RTS) game benchmarks—namely, their narrow evaluation scope and static environments. To this end, we introduce RTSGameBench, the first systematic evaluation benchmark built upon the large-scale RTS game Beyond All Reason. It features dynamically generated, diverse mini-games and a scalable, self-evolving framework to comprehensively assess models’ strategic abilities in multi-agent coordination and task scalability. We also develop RTSGameAgent, which integrates finite state machines with agent memory mechanisms to enable effective unit control in large-scale battlefields. Experiments reveal that state-of-the-art models exhibit significant deficiencies in tightly coupled collaboration and scalability tasks, underscoring the challenge and efficacy of our benchmark.

Abstract

Modern Vision-Language Models (VLMs) often struggle with strategic reasoning, i.e., anticipating and influencing other agents' actions, under uncertainty in competitive and cooperative settings. Real-time strategy (RTS) games can be a natural testbed for diagnosing this limitation, as they demand coordination with allies, adaptation to opponents' strategy, and long-horizon planning under partial observability. However, existing RTS benchmarks offer limited evaluation scope, lack systematic competency diagnosis, and remain fixed in the pre-designed scenario coverage. To address these limitations, we present RTSGameBench, which is built on Beyond All Reason, a large-scale RTS game with an expanded battlefield that demands broader strategy diversity than the existing testbeds. The proposed benchmark provides evaluations through diverse gameplay across various matchup structures, diagnostic assessment via mini-games, each targeting an individual strategic competency, and extensible coverage via a self-evolving generation framework that converts free-form queries into new mini-games, improving over successive cycles. Additionally, for VLMs to operate in large-scale RTS games, we provide RTSGameAgent that manages units by an FSM with agentic memory. We empirically validate that multiple state-of-the-art VLMs do not perform well when matchups demand tighter coordination, multiagent coordination and when task scale increases.

Problem

• strategic reasoning
• vision-language models
• real-time strategy games
• partial observability
• multiagent coordination

Innovation

• strategic reasoning
• vision-language models
• real-time strategy games
• self-evolving benchmark
• agent coordination