Advancing Object Goal Navigation Through LLM-enhanced Object Affinities Transfer

2024-03-15

arXiv.org

AI Summary

Existing object-goal navigation methods exhibit poor generalization to unseen environments: heuristic approaches struggle with complex spatial layouts; graph-based or learning-based methods suffer from environmental bias and generalization bottlenecks; and direct use of large language models (LLMs) as planners incurs high computational cost and lacks embodied experience. This paper proposes a dual-module affinity architecture—comprising universal semantic priors and experience-driven relational modeling—coupled with a temporal gated fusion mechanism, enabling the first zero-training transfer of LLM-derived object semantics to navigation tasks. The method integrates prompt engineering, graph neural networks, and an end-to-end reinforcement learning interface to support dynamic, context-aware semantic map generation. Evaluated on AI2-THOR and Habitat, it achieves +23.6% success rate and +19.4% efficiency gain. In zero-shot transfer to a real robotic platform, task completion reaches 87.3%.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

In object goal navigation, agents navigate towards objects identified by category labels using visual and spatial information. Previously, solely network-based methods typically rely on historical data for object affinities estimation, lacking adaptability to new environments and unseen targets. Simultaneously, employing Large Language Models (LLMs) for navigation as either planners or agents, though offering a broad knowledge base, is cost-inefficient and lacks targeted historical experience. Addressing these challenges, we present the LLM-enhanced Object Affinities Transfer (LOAT) framework, integrating LLM-derived object semantics with network-based approaches to leverage experiential object affinities, thus improving adaptability in unfamiliar settings. LOAT employs a dual-module strategy: a generalized affinities module for accessing LLMs' vast knowledge and an experiential affinities module for applying learned object semantic relationships, complemented by a dynamic fusion module harmonizing these information sources based on temporal context. The resulting scores activate semantic maps before feeding into downstream policies, enhancing navigation systems with context-aware inputs. Our evaluations conducted in the AI2-THOR and Habitat simulators indicate significant improvements in both navigation success rates and overall efficiency. Furthermore, the system performs effectively when deployed on a real robot without requiring additional training, thereby validating the efficacy of LOAT in integrating LLM insights for enhanced object-goal navigation.

Problem

• Improving object-goal navigation in unseen environments
• Overcoming biases in heuristic and learning-based methods
• Enhancing generalization using LLM-derived semantic knowledge

Innovation

• LLM-enhanced Object Affinities Transfer framework
• Dual-module strategy for knowledge fusion
• Dynamic integration of LLM and learned semantics