Can Vision-Language Models Solve the Shell Game?

2026-03-09

AI Summary

This work addresses the challenge that existing vision-language models struggle to track visually identical objects in videos due to their overreliance on static frame features and lack of explicit modeling for cross-temporal entity consistency. The authors introduce VET-Bench, a novel evaluation benchmark that theoretically demonstrates, for the first time, the inability of fixed-depth Transformers—without intermediate supervision—to distinguish indistinguishable objects. To overcome this limitation, they propose SGCoT, an end-to-end, tool-free spatiotemporal grounding chain-of-thought reasoning framework. SGCoT integrates synthetic data fine-tuning with the Molmo2 model to enable explicit trajectory reasoning grounded purely in textual alignment. Evaluated on VET-Bench, the method achieves over 90% accuracy, substantially outperforming current models and approaching human-level performance on video “shell game” tasks.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Language and Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Visual entity tracking is an innate cognitive ability in humans, yet it remains a critical bottleneck for Vision-Language Models (VLMs). This deficit is often obscured in existing video benchmarks by visual shortcuts. We introduce VET-Bench, a synthetic diagnostic testbed featuring visually identical objects that necessitate tracking exclusively through spatiotemporal continuity. Our experiments reveal that current state-of-the-art VLMs perform at or near chance level on VET-Bench, exposing a fundamental limitation: an over-reliance on static frame-level features and a failure to maintain entity representations over time. We provide a theoretical analysis drawing connections to the state-tracking problem, proving that fixed-depth transformer-based VLMs are fundamentally limited in tracking indistinguishable objects without intermediate supervision due to expressivity constraints. To address this, we propose Spatiotemporal Grounded Chain-of-Thought (SGCoT): generating object trajectories as explicit intermediate states. Leveraging Molmo2's object tracking ability, we elicit SGCoT reasoning by fine-tuning on synthesized text-only data for alignment. Our method achieves state-of-the-art accuracy exceeding 90% on VET-Bench, demonstrating that VLMs can reliably solve the video shell-game task end-to-end without external tools. Our code and data are available at https://vetbench.github.io .

Problem

• visual entity tracking
• vision-language models
• spatiotemporal continuity
• indistinguishable objects
• state-tracking

Innovation

• Vision-Language Models
• Visual Entity Tracking
• Spatiotemporal Reasoning
• Chain-of-Thought
• Synthetic Benchmark