SportD: Can VLMs Physically Strategize?

2026-07-16

AI Summary

This study investigates whether vision-language models (VLMs) can make effective physical strategic decisions grounded in real-world visual scenes, specifically focusing on optimal action selection—shooting or passing—in football matches. To this end, we introduce SportD, a benchmark comprising 478 offensive scenarios from the 2022 FIFA World Cup, and propose a novel quantitative evaluation framework centered on action values, integrating possession value modeling with counterfactual analysis to assess VLMs’ strategic reasoning capabilities. Experimental results reveal that even the best-performing VLM selects the highest-value action in only 31.4% of scenarios, lagging behind professional players at 38.9%. Moreover, VLMs consistently favor low-risk, conservative strategies, resulting in higher regret, which suggests a tendency to imitate observed human behavior rather than perform rational counterfactual evaluation.

Abstract

Vision--language models have become increasingly capable of interpreting visual scenes, but it remains unclear whether they can use information to make strategically effective decisions. We investigate this question in soccer, where models observe the seconds preceding an on-ball decision and must choose whether to shoot or pass to a specific teammate. Unlike conventional visual-understanding tasks, soccer enables decisions to be evaluated quantitatively by estimating the value of every available action. We introduce SportD, a benchmark comprising 478 on-ball decisions from the 2022 FIFA World Cup. Each model choice is evaluated against a possession-value model that estimates the action that most increases the attacking team's probability of scoring, allowing us to measure both optimal-action accuracy and the value forfeited by suboptimal decisions. Across three frontier VLMs, the best selects the highest-valued action on 31.4% of events, compared with 38.9% for the professional players, and all models incur significantly greater regret. Further analysis reveals a systematic preference for lower-variance and lower-reward actions: VLMs shoot less often and select substantially less progressive passes than either the optimal policy or the real players. The models also reproduce the player's specific action above chance even when that action is suboptimal, suggesting partial imitation of familiar play patterns rather than consistent evaluation of counterfactual alternatives. SportD provides a value-grounded testbed for measuring physical strategic reasoning in VLMs.

Problem

• vision-language models
• strategic reasoning
• sports decision-making
• action evaluation
• physical reasoning

Innovation

• vision-language models
• strategic reasoning
• sports AI
• possession-value model
• decision benchmarking