Prompt-Guided Spatial Understanding with RGB-D Transformers for Fine-Grained Object Relation Reasoning

2025-10-13

AI Summary

To address the weak spatial reasoning capability and poor generalization of vision-language models in large-scale 3D warehouse scenes—caused by severe occlusion and cluttered object layouts—this paper proposes a structured prompt-enhancement framework. It explicitly embeds normalized bounding-box coordinates into multimodal prompts and integrates an RGB-D Transformer architecture with a mask-dimension-guided spatial localization mechanism. Fine-grained spatial relationship modeling is achieved through joint vision-language learning, enabled by task-specific supervision, geometry-aware prompt engineering, and normalized answer injection. Evaluated on the AI City Challenge Track 3 benchmark, the method achieves a total score of 73.0606 (ranked fourth), demonstrating significant improvements in spatial reasoning accuracy and evaluation consistency within complex industrial environments.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Computer Vision: 3D Computer Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Spatial reasoning in large-scale 3D environments such as warehouses remains a significant challenge for vision-language systems due to scene clutter, occlusions, and the need for precise spatial understanding. Existing models often struggle with generalization in such settings, as they rely heavily on local appearance and lack explicit spatial grounding. In this work, we introduce a dedicated spatial reasoning framework for the Physical AI Spatial Intelligence Warehouse dataset introduced in the Track 3 2025 AI City Challenge. Our approach enhances spatial comprehension by embedding mask dimensions in the form of bounding box coordinates directly into the input prompts, enabling the model to reason over object geometry and layout. We fine-tune the framework across four question categories namely: Distance Estimation, Object Counting, Multi-choice Grounding, and Spatial Relation Inference using task-specific supervision. To further improve consistency with the evaluation system, normalized answers are appended to the GPT response within the training set. Our comprehensive pipeline achieves a final score of 73.0606, placing 4th overall on the public leaderboard. These results demonstrate the effectiveness of structured prompt enrichment and targeted optimization in advancing spatial reasoning for real-world industrial environments.

Problem

• Enhancing spatial reasoning in cluttered 3D warehouse environments
• Improving object relation understanding through geometry-aware prompt embedding
• Addressing generalization challenges in vision-language systems for industrial settings

Innovation

• Embedding bounding box coordinates into input prompts
• Fine-tuning RGB-D transformers for spatial reasoning tasks
• Appending normalized answers to GPT responses for consistency