A Spatial Relationship Aware Dataset for Robotics

2025-06-14

AI Summary

Accurate modeling of fine-grained spatial relationships among objects—such as co-occurrence of visually similar objects and multi-level spatial nesting—is critical for real-world robotic task planning in complex indoor environments; however, existing datasets lack 3D positional annotations and explicit spatial relation labels. To address this gap, we introduce SpotSRD, the first spatial-relation-aware dataset designed for real-robot deployment, comprising nearly 1,000 indoor images captured by Boston Dynamics’ Spot robot, with fine-grained annotations of object attributes, 3D coordinates, and 24 spatial relation types. We develop a custom annotation tool and a scene-graph evaluation framework, systematically exposing spatial modeling bottlenecks across six state-of-the-art models. Furthermore, we propose a spatially aware prompting mechanism that injects structured spatial relations into GPT-4o, significantly improving its accuracy on spatial planning tasks. The dataset, annotation tool, and code are publicly released.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Robotic task planning in real-world environments requires not only object recognition but also a nuanced understanding of spatial relationships between objects. We present a spatial-relationship-aware dataset of nearly 1,000 robot-acquired indoor images, annotated with object attributes, positions, and detailed spatial relationships. Captured using a Boston Dynamics Spot robot and labelled with a custom annotation tool, the dataset reflects complex scenarios with similar or identical objects and intricate spatial arrangements. We benchmark six state-of-the-art scene-graph generation models on this dataset, analysing their inference speed and relational accuracy. Our results highlight significant differences in model performance and demonstrate that integrating explicit spatial relationships into foundation models, such as ChatGPT 4o, substantially improves their ability to generate executable, spatially-aware plans for robotics. The dataset and annotation tool are publicly available at https://github.com/PengPaulWang/SpatialAwareRobotDataset, supporting further research in spatial reasoning for robotics.

Problem

• Enhancing robotic task planning with spatial relationship understanding
• Evaluating scene-graph models for spatial reasoning accuracy and speed
• Improving foundation models' ability to generate spatially-aware robotic plans

Innovation

• Spatial-relationship-aware dataset with robot-acquired images
• Custom annotation tool for detailed spatial relationships
• Integration of spatial relationships into foundation models