Is Visual Prompting All You Need? Studying VLM Spatial Reasoning under Progressive Visual Scaffolds

2026-08-21

AI Summary

研究通过引入轻量级视觉支架改善视觉-语言模型在空间推理任务中的表现，减少基础相关错误，提高任务准确性。

Abstract

Vision-language models (VLMs) have advanced rapidly in multimodal reasoning, yet recent work shows that their failures often reflect an interaction between visual grounding and downstream reasoning. What remains less clear is how the visual presentation of a task shapes model performance and failure modes when the underlying reasoning problem is unchanged. We study this question in SPaRC, a benchmark for grid-based visual spatial planning, by introducing lightweight input-side scaffolds that preserve the visual modality while making spatial structure more accessible. Across multiple VLMs, these scaffolds improve task accuracy over the original visual setting by up to 34.0 percentage points and further complement GRPO-based training, yielding up to 4.6 additional accuracy points compared with near-zero gains on the original visual input. Analyses on both end-to-end task solving and object detection show that these gains are closely tied to reductions in grounding-related errors, while rule reasoning remains comparatively challenging. We find that visual presentation is a central factor that determines whether VLM benchmarks measure grounded perception, downstream reasoning, or a mixture of both.

Problem

• Vision-language models
• Spatial reasoning
• Visual presentation
• Grounded perception

Innovation

• visual scaffolds
• spatial reasoning
• grounding-related errors
• vision-language models