GamiBench: Evaluating Spatial Reasoning and 2D-to-3D Planning Capabilities of MLLMs with Origami Folding Tasks

2025-12-21

AI Summary

Current multimodal large language models (MLLMs) lack standardized evaluation for spatial reasoning and 2D-to-3D planning capabilities—particularly in geometrically rigorous, physically constrained tasks. Method: We introduce GamiBench, the first dedicated benchmark for origami folding, comprising 186 pairs of foldable/unfoldable 2D crease patterns and their corresponding multi-view 3D folded configurations, synthetically generated via computational origami geometry. We propose two novel diagnostic metrics: View Consistency (VC) and Impossible-Fold Recognition Rate (IFSR), enabling systematic assessment of dynamic folding reasoning, physical feasibility judgment, and cross-view verification. Contribution/Results: Tasks include 3D configuration prediction, valid-view identification, and impossible-pattern detection. Experiments reveal significant deficiencies in state-of-the-art models—including GPT-5 and Gemini-2.5-Pro—in single-step spatial understanding. The dataset and evaluation framework are open-sourced to advance standardized assessment of MLLMs’ geometric cognition.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Multimodal large language models (MLLMs) are proficient in perception and instruction-following, but they still struggle with spatial reasoning: the ability to mentally track and manipulate objects across multiple views and over time. Spatial reasoning is a key component of human intelligence, but most existing benchmarks focus on static images or final outputs, failing to account for the sequential and viewpoint-dependent nature of this skill. To close this gap, we introduce GamiBench, a benchmark designed to evaluate spatial reasoning and 2D-to-3D planning in MLLMs through origami-inspired folding tasks. GamiBench includes 186 regular and 186 impossible 2D crease patterns paired with their corresponding 3D folded shapes, produced from six distinct viewpoints across three visual question-answering (VQA) tasks: predicting 3D fold configurations, distinguishing valid viewpoints, and detecting impossible patterns. Unlike previous benchmarks that assess only final predictions, GamiBench holistically evaluates the entire reasoning process--measuring cross-view consistency, physical feasibility through impossible-fold detection, and interpretation of intermediate folding steps. It further introduces new diagnostic metrics--viewpoint consistency (VC) and impossible fold selection rate (IFSR)--to measure how well models handle folds of varying complexity. Our experiments show that even leading models such as GPT-5 and Gemini-2.5-Pro struggle on single-step spatial understanding. These contributions establish a standardized framework for evaluating geometric understanding and spatial reasoning in MLLMs. Dataset and code: https://github.com/stvngo/GamiBench.

Problem

• Evaluates spatial reasoning and 2D-to-3D planning in multimodal language models.
• Assesses models on origami folding tasks across multiple viewpoints and steps.
• Measures cross-view consistency and detection of physically impossible folds.

Innovation

• Benchmark for spatial reasoning with origami tasks
• Measures cross-view consistency and physical feasibility
• Introduces new diagnostic metrics for model evaluation