Imaginative World Modeling with Scene Graphs for Embodied Agent Navigation

2025-08-09

AI Summary

To address low semantic navigation efficiency of embodied agents in unknown environments, this paper proposes the Imagination-Guided Scene Graph World Model (IG-WM). IG-WM integrates symbolic scene graphs with large language models (LLMs) to construct a hierarchical, global semantic representation of the environment, enabling cross-room and cross-floor semantic reasoning and proactive prediction. It further introduces semantic shortcut identification and adaptive exploration strategies, reducing reliance on historical observations. Evaluated on HM3D and HSSD benchmarks, IG-WM achieves task success rates of 65.4% and 66.8%, respectively—substantially outperforming prior methods—and demonstrates robust cross-level navigation capability in real-world settings. This work pioneers a novel world modeling paradigm for embodied intelligence by unifying interpretable symbolic structures with LLM-driven imagination, yielding models that are both generalizable and logically verifiable.

Technology Category

• Intelligent Robots: Embodied AI
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Symbolic Representations

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Semantic navigation requires an agent to navigate toward a specified target in an unseen environment. Employing an imaginative navigation strategy that predicts future scenes before taking action, can empower the agent to find target faster. Inspired by this idea, we propose SGImagineNav, a novel imaginative navigation framework that leverages symbolic world modeling to proactively build a global environmental representation. SGImagineNav maintains an evolving hierarchical scene graphs and uses large language models to predict and explore unseen parts of the environment. While existing methods solely relying on past observations, this imaginative scene graph provides richer semantic context, enabling the agent to proactively estimate target locations. Building upon this, SGImagineNav adopts an adaptive navigation strategy that exploits semantic shortcuts when promising and explores unknown areas otherwise to gather additional context. This strategy continuously expands the known environment and accumulates valuable semantic contexts, ultimately guiding the agent toward the target. SGImagineNav is evaluated in both real-world scenarios and simulation benchmarks. SGImagineNav consistently outperforms previous methods, improving success rate to 65.4 and 66.8 on HM3D and HSSD, and demonstrating cross-floor and cross-room navigation in real-world environments, underscoring its effectiveness and generalizability.

Problem

• Enables agents to navigate unseen environments using predictive scene modeling
• Leverages symbolic world modeling for proactive environmental representation
• Improves navigation success rates through adaptive semantic exploration

Innovation

• Symbolic world modeling with hierarchical scene graphs
• LLM-predicted exploration of unseen environment parts
• Adaptive navigation using semantic shortcuts