MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents

2025-05-26

AI Summary

Existing multimodal large language models (MLLMs) are evaluated on spatial intelligence primarily through abstract visual question answering, failing to assess embodied spatial planning capabilities required in open-world environments. Method: We introduce MineAnyBuild, the first benchmark for spatial planning tailored to AI agents, grounded in Minecraft. It requires models to generate executable architectural plans from multimodal human instructions. The benchmark extends evaluation beyond abstract reasoning to embodied task execution, establishing a four-dimensional assessment framework covering spatial understanding, reasoning, creativity, and commonsense knowledge. It supports infinite, player-generated data expansion and integrates MLLMs, game engine interfaces, procedural task generation, and structured evaluation protocols. Results: Comprehensive evaluation reveals significant spatial planning deficiencies in current MLLM-based agents, confirming the benchmark’s sensitivity to capability disparities and its scalability. MineAnyBuild provides a standardized, extensible testbed for advancing spatial intelligence research.

Technology Category

• Multiagent Systems: Multiagent Planning
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Spatial Planning is a crucial part in the field of spatial intelligence, which requires the understanding and planning about object arrangements in space perspective. AI agents with the spatial planning ability can better adapt to various real-world applications, including robotic manipulation, automatic assembly, urban planning etc. Recent works have attempted to construct benchmarks for evaluating the spatial intelligence of Multimodal Large Language Models (MLLMs). Nevertheless, these benchmarks primarily focus on spatial reasoning based on typical Visual Question-Answering (VQA) forms, which suffers from the gap between abstract spatial understanding and concrete task execution. In this work, we take a step further to build a comprehensive benchmark called MineAnyBuild, aiming to evaluate the spatial planning ability of open-world AI agents in the Minecraft game. Specifically, MineAnyBuild requires an agent to generate executable architecture building plans based on the given multi-modal human instructions. It involves 4,000 curated spatial planning tasks and also provides a paradigm for infinitely expandable data collection by utilizing rich player-generated content. MineAnyBuild evaluates spatial planning through four core supporting dimensions: spatial understanding, spatial reasoning, creativity, and spatial commonsense. Based on MineAnyBuild, we perform a comprehensive evaluation for existing MLLM-based agents, revealing the severe limitations but enormous potential in their spatial planning abilities. We believe our MineAnyBuild will open new avenues for the evaluation of spatial intelligence and help promote further development for open-world AI agents capable of spatial planning.

Problem

• Evaluating spatial planning in open-world AI agents using Minecraft
• Assessing spatial understanding, reasoning, creativity, and commonsense in agents
• Bridging abstract spatial reasoning with concrete task execution

Innovation

• Benchmark for spatial planning in Minecraft
• 4,000 tasks with expandable data collection
• Evaluates spatial understanding, reasoning, creativity, commonsense