IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoning

2026-07-03

AI Summary

This work addresses the lack of effective evaluation of holistic 3D spatial reasoning capabilities in current vision-language models. It introduces the first diagnostic benchmark specifically designed to assess geometric and structural understanding, moving beyond conventional question-answering or simplistic geometric reconstruction paradigms. The benchmark leverages 1,000 procedurally generated, photorealistic indoor scenes created in Blender, offering semantic fidelity, controllable lighting and viewpoints, and asset interchangeability. Model outputs are evaluated through multidimensional metrics and render-based comparisons. Comprehensive evaluation across 15 state-of-the-art models reveals a significant performance gap: even the best model achieves only 62.1 out of 100 points, highlighting a pronounced asymmetry between geometric reasoning and object recognition capabilities.

Abstract

Spatial question answering is the dominant paradigm for evaluating spatial intelligence in Vision-Language Models (VLMs), but it leaves a complementary axis of spatial competence under-evaluated: holistic 3D layout inference, which predicts every visible object's pose and extent from a single image in a structured form. To this end, we introduce IDEAL-Bench, an evaluation suite that requires VLMs to predict structured 3D layouts on photorealistic indoor scenes across 10 room types, scored along five numerical dimensions and a perceptual render-and-compare protocol. By operating on semantically realistic scenes with full asset substitution under controlled lighting and viewpoint, IDEAL-Bench moves beyond CLEVR-style simple geometric primitives so that any image-space discrepancy reflects spatial reasoning alone. The benchmark is built on IDEAL-Scenes, a procedurally generated dataset of 1,000 re-renderable Blender environments with ground-truth layouts. Evaluating 15 prominent VLMs reveals three findings: the task remains substantially unsolved, with the strongest model reaching only 62.1/100 overall; all models exhibit a sharp asymmetry between object recognition and geometric regression, indicating that current VLMs are trained to describe scenes rather than to measure them; model rankings partially diverge from those on QA-based and primitive-reconstruction benchmarks: top-tier consensus holds, but mid-tier rankings shift substantially. Collectively, these findings establish IDEAL-Bench as a diagnostic suite, targeting the geometric and structural competencies that QA-based evaluation cannot surface, and paving the way towards more rigorous evaluation of spatial intelligence in next-generation VLMs. Together, these findings position IDEAL-Bench as a principled diagnostic for whether future VLMs achieve genuine spatial understanding rather than linguistic approximations of it.

Problem

• 3D layout inference
• spatial reasoning
• Vision-Language Models
• spatial intelligence evaluation
• structured scene understanding

Innovation

• 3D layout inference
• spatial reasoning benchmark
• procedurally generated scenes
• vision-language models
• render-and-compare evaluation