SpatialPrompting: Keyframe-driven Zero-Shot Spatial Reasoning with Off-the-Shelf Multimodal Large Language Models

2025-05-08

AI Summary

Existing 3D spatial reasoning methods rely on 3D-specific inputs (e.g., point clouds or voxels) or require costly 3D fine-tuning of multimodal models, limiting generalizability and scalability. Method: We propose a zero-shot 3D spatial reasoning paradigm that leverages off-the-shelf multimodal large language models (MLLMs, e.g., LLaVA, Qwen-VL) exclusively from 2D images—without 3D fine-tuning or geometric inputs. Key-frame selection integrates visual-linguistic similarity, Mahalanobis distance, field-of-view, and image sharpness to extract representative views and their camera poses; pose embeddings are then fused into MLLM prompts to enable 2D-image–driven spatial relation modeling. Contribution/Results: This is the first work to achieve fully zero-shot, fine-tuning-free, and purely 2D-input 3D spatial reasoning. It establishes new state-of-the-art zero-shot performance on ScanQA and SQA3D, significantly outperforming prior 3D-fine-tuned approaches. Our method offers a lightweight, generalizable, and scalable pathway toward embodied spatial understanding.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This study introduces SpatialPrompting, a novel framework that harnesses the emergent reasoning capabilities of off-the-shelf multimodal large language models to achieve zero-shot spatial reasoning in three-dimensional (3D) environments. Unlike existing methods that rely on expensive 3D-specific fine-tuning with specialized 3D inputs such as point clouds or voxel-based features, SpatialPrompting employs a keyframe-driven prompt generation strategy. This framework uses metrics such as vision-language similarity, Mahalanobis distance, field of view, and image sharpness to select a diverse and informative set of keyframes from image sequences and then integrates them with corresponding camera pose data to effectively abstract spatial relationships and infer complex 3D structures. The proposed framework not only establishes a new paradigm for flexible spatial reasoning that utilizes intuitive visual and positional cues but also achieves state-of-the-art zero-shot performance on benchmark datasets, such as ScanQA and SQA3D, across several metrics. The proposed method effectively eliminates the need for specialized 3D inputs and fine-tuning, offering a simpler and more scalable alternative to conventional approaches.

Problem

• Enables zero-shot 3D spatial reasoning without fine-tuning
• Uses keyframe selection to infer complex 3D structures
• Eliminates need for specialized 3D inputs like point clouds

Innovation

• Keyframe-driven prompt generation for spatial reasoning
• Uses vision-language similarity and camera pose data
• Achieves zero-shot performance without 3D fine-tuning