Graph2Nav: 3D Object-Relation Graph Generation to Robot Navigation

2025-04-23

AI Summary

To address the low efficiency of robot autonomous navigation—particularly object search—in real-world environments, this paper proposes the first end-to-end 3D object-relation graph generation framework that requires no 3D relational annotations. Methodologically, it extends 2D panoptic scene graph techniques to 3D semantic mapping, constructing a hierarchical and semantically rich 3D scene graph; it integrates a 2D scene graph model, real-time 3D perception, large language model–driven navigation planning (SayNav), and a robotic system. The key contribution is the first end-to-end learning of 3D semantic relations without scarce, manually annotated 3D relational data—overcoming a critical bottleneck in prior approaches reliant on costly supervision. Experiments demonstrate state-of-the-art accuracy in 3D object localization and relation labeling, and achieve a 37% improvement in object search success rate in real-world settings.

Technology Category

• Intelligent Robots: Localization, Mapping, and Navigation
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

We propose Graph2Nav, a real-time 3D object-relation graph generation framework, for autonomous navigation in the real world. Our framework fully generates and exploits both 3D objects and a rich set of semantic relationships among objects in a 3D layered scene graph, which is applicable to both indoor and outdoor scenes. It learns to generate 3D semantic relations among objects, by leveraging and advancing state-of-the-art 2D panoptic scene graph works into the 3D world via 3D semantic mapping techniques. This approach avoids previous training data constraints in learning 3D scene graphs directly from 3D data. We conduct experiments to validate the accuracy in locating 3D objects and labeling object-relations in our 3D scene graphs. We also evaluate the impact of Graph2Nav via integration with SayNav, a state-of-the-art planner based on large language models, on an unmanned ground robot to object search tasks in real environments. Our results demonstrate that modeling object relations in our scene graphs improves search efficiency in these navigation tasks.

Problem

• Generating real-time 3D object-relation graphs for robot navigation
• Learning 3D semantic relations from 2D panoptic scene graphs
• Improving object search efficiency via scene graph modeling

Innovation

• Real-time 3D object-relation graph generation
• Leverages 2D panoptic scene graphs in 3D
• Integrates with LLM-based planners for navigation