🤖 AI Summary
Existing spatial reasoning benchmarks (e.g., CSR) emphasize textual comprehension while neglecting executable path planning in embodied environments. Method: We introduce GRASP—a novel, large-scale grid-world benchmark (16,000 scenarios) for embodied spatial planning—systematically controlling energy constraints, obstacle configurations, agent starting positions, and motion restrictions to enable fine-grained, reproducible evaluation of commonsense spatial reasoning. For the first time, we deploy LLMs in actionable grid-based planning tasks, comparing GPT-3.5-Turbo, GPT-4o, and GPT-o1-mini against classical baselines including random walk and greedy search. Contribution/Results: Empirical results reveal that even state-of-the-art LLMs struggle to consistently generate valid, executable plans on GRASP, exposing fundamental limitations in their capacity for embodied spatial planning—highlighting a critical gap between linguistic understanding and grounded, executable reasoning.
📝 Abstract
Spatial reasoning, an important faculty of human cognition with many practical applications, is one of the core commonsense skills that is not purely language-based and, for satisfying (as opposed to optimal) solutions, requires some minimum degree of planning. Existing benchmarks of Commonsense Spatial Reasoning (CSR) tend to evaluate how Large Language Models (LLMs) interpret text-based spatial $ extit{descriptions}$ rather than directly evaluate a plan produced by the LLM in response to a $ extit{specific}$ spatial reasoning problem. In this paper, we construct a large-scale benchmark called GRASP, which consists of 16,000 grid-based environments where the agent is tasked with an energy collection problem. These environments include 100 grid instances instantiated using each of the 160 different grid settings, involving five different energy distributions, two modes of agent starting position, and two distinct obstacle configurations, as well as three kinds of agent constraints. Using GRASP, we compare classic baseline approaches, such as random walk and greedy search methods, with advanced LLMs like GPT-3.5-Turbo, GPT-4o, and GPT-o1-mini. The experimental results indicate that even these advanced LLMs struggle to consistently achieve satisfactory solutions.