SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Models

2025-04-25

AI Summary

3D referring expression comprehension faces challenges including scene diversity, fine-grained object abundance, high linguistic variability, and scarcity of 3D annotated data. To address these, we propose the first zero-shot 3D referring grounding framework that requires no 3D textual annotations. Our method leverages 2D–3D cross-modal attribute transfer to acquire object semantics and integrates large language model (LLM)-driven multi-step spatial semantic reasoning with an interpretable, heuristic spatial rule engine for viewpoint-dependent precise localization. The framework enables zero-shot generalization to unseen environments and has been successfully deployed in real time on an embedded automotive platform. It achieves state-of-the-art performance on two mainstream benchmarks and demonstrates practical efficacy by enabling a robot to navigate to target objects in previously unseen real-world scenes.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Interpreting object-referential language and grounding objects in 3D with spatial relations and attributes is essential for robots operating alongside humans. However, this task is often challenging due to the diversity of scenes, large number of fine-grained objects, and complex free-form nature of language references. Furthermore, in the 3D domain, obtaining large amounts of natural language training data is difficult. Thus, it is important for methods to learn from little data and zero-shot generalize to new environments. To address these challenges, we propose SORT3D, an approach that utilizes rich object attributes from 2D data and merges a heuristics-based spatial reasoning toolbox with the ability of large language models (LLMs) to perform sequential reasoning. Importantly, our method does not require text-to-3D data for training and can be applied zero-shot to unseen environments. We show that SORT3D achieves state-of-the-art performance on complex view-dependent grounding tasks on two benchmarks. We also implement the pipeline to run real-time on an autonomous vehicle and demonstrate that our approach can be used for object-goal navigation on previously unseen real-world environments. All source code for the system pipeline is publicly released at https://github.com/nzantout/SORT3D .

Problem

• Grounding 3D objects using language without training data
• Interpreting complex spatial relations in diverse scenes
• Zero-shot generalization to unseen real-world environments

Innovation

• Utilizes 2D object attributes for 3D grounding
• Combines heuristics-based spatial reasoning with LLMs
• Zero-shot generalization without text-to-3D training