Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Models

2025-06-30

AI Summary

This work investigates visual language models’ (VLMs) capacity to understand and reason about spatial object deformations (2D→3D), systematically evaluating their performance on forward reasoning (predicting outcomes from given operations) and inverse reasoning (inferring operations from given outcomes). To this end, we introduce the first benchmark dedicated to spatial deformation reasoning, built upon a novel “ladder”-style evaluation framework. This framework leverages an automated data engine to generate leakage-free, scalable-step problem pairs and adopts a step-wise difficulty grading scheme. It enables the first systematic characterization of VLMs’ spatial reasoning capability boundaries. Extensive experiments reveal that state-of-the-art VLMs—including those fine-tuned specifically for spatial tasks or enhanced with advanced inference techniques—exhibit severely limited performance on these 3D spatial reasoning tasks, exposing fundamental deficiencies in their geometric understanding and compositional spatial reasoning abilities.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Computer Vision: Visual Reasoning & Symbolic Representations
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Humans naturally possess the spatial reasoning ability to form and manipulate images and structures of objects in space. There is an increasing effort to endow Vision-Language Models (VLMs) with similar spatial reasoning capabilities. However, it remains unclear whether these models truly understand and manipulate spatial objects or not. To address this question, we propose a new evaluation framework aimed at assessing the performance of VLMs in spatial deformation reasoning tasks. Specifically, we construct a benchmark for spatial deformation reasoning from 2D to 3D. Leveraging our data engine, we can generate unlimited evaluation problem pairs with infinite steps, without any data leakage. We explore whether the model can effectively perform spatial deformation reasoning from two directions: forward reasoning (given the operations, find the final state) and reverse reasoning (given the final state, determine the operations). We adopt a ladder competition format, using the number of deformation steps as the level classification criterion, with the goal of exploring the boundaries of the model's deformation reasoning capabilities. Interestingly, the benchmarking results reveal that almost no model demonstrates plausible spatial deformation reasoning abilities. Furthermore, even after applying targeted training and mainstream reasoning enhancement methods, the models are still unable to perform well on 3D spatial deformation reasoning.

Problem

• Assessing VLMs' spatial deformation reasoning abilities
• Evaluating forward and reverse 2D-to-3D deformation reasoning
• Testing model limits with infinite-step ladder competition

Innovation

• Propose a new spatial deformation reasoning benchmark
• Generate unlimited evaluation pairs without data leakage
• Adopt ladder competition to explore reasoning boundaries