ComplexityWorld: Benchmarking Vision-Language Models on Verifiable Visual Decision Making

2026-08-05

AI Summary

Current vision-language models struggle with structured decision-making tasks that require grounding in visual evidence while satisfying global constraints. To address this gap, this work introduces ComplexityWorld—the first systematic benchmark comprising 390 procedurally generated visual decision tasks derived from hidden structural specifications, spanning 39 domains and 29 problem types, each evaluated for correctness via executable verifiers. Leveraging programmatic generation, multimodal reasoning evaluation, and an agent-based framework, the benchmark exposes a fundamental bottleneck in existing models: their inability to reliably translate implicit visual information into explicit structured decisions. Except for GPT-5.6-Sol, which achieves a 75.6% verification pass rate, all other models score below 40%. Performance markedly improves when decision-relevant information is provided explicitly in structured form, underscoring current models’ limited capacity to infer and reason over implicit constraints embedded in visual scenes.

Abstract

Vision-language models (VLMs) have made rapid progress in visual perception and increasingly support real-world tasks that depend on images. Many such tasks, however, require more than rec- ognizing what an image contains: a model must use visual evidence to make a complete decision whose parts jointly satisfy global constraints. We introduce COMPLEXITYWORLD, a benchmark of 390 tasks across 39 domain-inspired visual worlds and 29 decision categories. Each task is generated from a hidden structured specification, rendered as a visual scene, and scored by an exe- cutable verifier that accepts any feasible solution. Under direct inference, all evaluated models ex- cept GPT-5.6-Sol remain below 40% verifier ac- ceptance rate (VAR), while GPT-5.6-Sol reaches 75.6%. Performance improves substantially when the same decision information is made explicit in structured form, yet varies sharply across equiva- lent visual presentations. Agent scaffolds provide smaller, model-dependent gains. Together, these results reveal a persistent visual-to-decision bot- tleneck that additional inference alone does not remove.

Problem

• vision-language models
• visual decision making
• verifiable reasoning
• global constraints
• visual perception

Innovation

• vision-language models
• verifiable decision making
• executable verifier
• structured reasoning
• visual reasoning benchmark