MindEdit-Bench: Benchmarking Object-Level Counterfactual Spatial Reasoning in VLMs from In-the-Wild Photos

2026-07-01

AI Summary

This work addresses the lack of effective evaluation for object-level counterfactual spatial reasoning in current vision-language models (VLMs), particularly their ability to predict scene changes following object movement or rotation. To this end, the authors introduce MindEdit-Bench, a benchmark constructed from triple-view smartphone images of 120 private indoor scenes, leveraging an automated 3D scene graph extraction pipeline to formulate six spatial reasoning tasks. Notably, levels L4 and L5 uniquely focus on object-level counterfactual reasoning involving entities not visible in the input. The benchmark employs a structured multiple-choice design enabling fine-grained error diagnosis and includes 1,003 human-validated questions. Evaluation of 15 prominent VLMs reveals accuracy rates of only 8%–31%, substantially below human performance (81%–97%), highlighting critical deficiencies in depth-axis reasoning and visibility-aware scene editing.

Abstract

Benchmarks for vision-language models (VLMs) mostly test observational spatial reasoning: models describe relations already visible in the input. Existing what-if tasks typically vary the observer while keeping the scene fixed. Can VLMs instead predict the consequences of hypothetically moving or rotating an object? We introduce MindEdit-Bench, a benchmark of six spatial reasoning tasks built from three-photo smartphone triplets of newly captured indoor scenes via an automatic in-the-wild 3D scene-graph extraction pipeline. Four tasks probe perception and perspective transformation over observed structure; two new tasks, L4 (spatial editing) and L5 (cross-view visibility editing), probe object-level counterfactual reasoning, where correct answers are absent from all input images. Each question provides 8-24 structured answer choices, enabling answer-letter-level diagnosis of spatial and fallback errors. The benchmark covers 120 private indoor scenes not drawn from public datasets, reducing public-data pretraining-overlap risk. Across 15 VLMs on 1,003 human-verified questions, task-wise mean VLM accuracy is only 8%-31%, versus 81%-97% human majority-vote accuracy. The pooled human--best-VLM gap is 53 pp, with at least 39 pp on every task. The structured answer space further reveals non-uniform failures, including weaker camera-depth-axis inference and fallback behavior on difficult visibility-editing cases.

Problem

• counterfactual reasoning
• spatial reasoning
• vision-language models
• object-level editing
• in-the-wild photos

Innovation

• counterfactual spatial reasoning
• object-level editing
• vision-language models
• 3D scene graph
• structured benchmark