GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generation

2025-10-24

AI Summary

Visual language models (VLMs) exhibit limited capability in spatial reasoning, primarily due to high error rates in existing data generation methods: 3D reconstruction from single images introduces cascading modeling errors (human verification rate: only 57.6%), while caption-based approaches suffer from hallucination and prohibitive annotation costs. To address this, we propose GRAID—the first framework that automatically generates high-quality spatial visual question-answering (VQA) pairs solely from standard 2D object detector outputs (bounding boxes), integrating geometric priors to directly model spatial relations without relying on error-prone 3D reconstruction or text generation. GRAID produces over 8.5 million samples on BDD100k, NuImages, and Waymo, achieving a human-verified accuracy of 91.16%. Fine-tuning Llama-3 (2B/11B) with GRAID-generated data yields up to a 47.5% absolute accuracy gain on unseen spatial reasoning tasks and establishes new state-of-the-art performance on benchmarks including BLINK.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Vision Language Models (VLMs) achieve strong performance on many vision-language tasks but often struggle with spatial reasoning extemdash{}a prerequisite for many applications. Empirically, we find that a dataset produced by a current training data generation pipeline has a 57.6% human validation rate. These rates stem from current limitations: single-image 3D reconstruction introduces cascading modeling errors and requires wide answer tolerances, while caption-based methods require hyper-detailed annotations and suffer from generative hallucinations. We present GRAID, built on the key insight that qualitative spatial relationships can be reliably determined from 2D geometric primitives alone. By operating exclusively on 2D bounding boxes from standard object detectors, GRAID avoids both 3D reconstruction errors and generative hallucinations, resulting in datasets that are of higher quality than existing tools that produce similar datasets as validated by human evaluations. We apply our framework to the BDD100k, NuImages, and Waymo datasets, generating over 8.5 million high-quality VQA pairs creating questions spanning spatial relations, counting, ranking, and size comparisons. We evaluate one of the datasets and find it achieves 91.16% human-validated accuracy extemdash{}compared to 57.6% on a dataset generated by recent work. % or recent work Critically, we demonstrate that when trained on GRAID data, models learn spatial reasoning concepts that generalize: models fine-tuned on 6 question types improve on over 10 held-out types, with accuracy gains of 47.5% on BDD and 37.9% on NuImages for Llama 3.2B 11B, and when trained on all questions types, achieve improvements on several existing benchmarks such as BLINK. The GRAID framework, datasets, and additional information can be found on our href{https://ke7.github.io/graid/}{project page}.

Problem

• Addressing VLMs' spatial reasoning limitations through high-fidelity data generation
• Overcoming 3D reconstruction errors and generative hallucinations in training data
• Generating high-quality spatial reasoning datasets to improve model generalization

Innovation

• Uses 2D bounding boxes for spatial reasoning
• Avoids 3D reconstruction errors and hallucinations
• Generates high-quality VQA pairs from datasets