ESPIRE: A Diagnostic Benchmark for Embodied Spatial Reasoning of Vision-Language Models

2026-03-13

AI Summary

This work addresses the limitations of existing vision-language models in embodied spatial reasoning evaluation, which suffers from overly simplistic paradigms and insufficient coverage, thereby hindering iterative development for real-world robotic tasks. To this end, the authors introduce ESPIRE, a diagnostic benchmark that reformulates embodied spatial reasoning as a generative task for the first time. By leveraging a physics-based simulation environment, ESPIRE decomposes robotic tasks into localization and execution subtasks and employs systematic scene and instruction design to move beyond traditional discriminative evaluation paradigms reliant on distractors. This approach enables fine-grained analysis—from passive comprehension to active intervention—effectively uncovering the strengths and weaknesses of mainstream vision-language models in embodied settings and providing clear guidance for future model improvements.

Technology Category

• Intelligent Robots: Embodied AI
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

A recent trend in vision-language models (VLMs) has been to enhance their spatial cognition for embodied domains. Despite progress, existing evaluations have been limited both in paradigm and in coverage, hindering rapid, iterative model development. To address these limitations, we propose ESPIRE, a diagnostic benchmark for embodied spatial reasoning. ESPIRE offers a simulated world that physically grounds VLMs and evaluates them on spatial-reasoning-centric robotic tasks, thus narrowing the gap between evaluation and real-world deployment. To adapt VLMs to robotic tasks, we decompose each task into localization and execution, and frame both as generative problems, in stark contrast to predominant discriminative evaluations (e.g., via visual-question answering) that rely on distractors and discard execution. This decomposition further enables a fine-grained analysis beyond passive spatial reasoning toward reasoning to act. We systematically design ESPIRE both at the instruction level and at the environment level, ensuring broad coverage of spatial reasoning scenarios. We use ESPIRE to diagnose a range of frontier VLMs and provide in-depth analysis of their spatial reasoning behaviors.

Problem

• embodied spatial reasoning
• vision-language models
• diagnostic benchmark
• robotic tasks
• spatial cognition

Innovation

• embodied spatial reasoning
• vision-language models
• generative evaluation
• robotic task decomposition
• diagnostic benchmark