GIST: Multimodal Knowledge Extraction and Spatial Grounding via Intelligent Semantic Topology

2026-04-16

AI Summary

This work addresses the challenges of semantic sparsity and spatial localization in dense environments such as retail stores, warehouses, and hospitals by proposing a lightweight multimodal semantic topology construction method. The approach integrates point clouds, 2D occupancy maps, and topological structure extraction, augmented with vision-language models (VLMs), large language models (LLMs), and intelligent keyframe-based semantic selection to generate semantically annotated navigation graphs. The system enables intent-driven semantic search, one-shot semantic localization, area classification, and natural language path generation. Experimental results demonstrate a Top-5 mean translational error of 1.04 meters in semantic localization, an 80% on-site navigation success rate using only voice commands, and significant outperformance over sequential instruction-generation baselines across multiple LLM evaluation metrics, thereby enhancing general accessibility in complex environments.

Technology Category

• Intelligent Robots: Localization, Mapping, and Navigation
• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Navigating complex, densely packed environments like retail stores, warehouses, and hospitals poses a significant spatial grounding challenge for humans and embodied AI. In these spaces, dense visual features quickly become stale given the quasi-static nature of items, and long-tail semantic distributions challenge traditional computer vision. While Vision-Language Models (VLMs) help assistive systems navigate semantically-rich spaces, they still struggle with spatial grounding in cluttered environments. We present GIST (Grounded Intelligent Semantic Topology), a multimodal knowledge extraction pipeline that transforms a consumer-grade mobile point cloud into a semantically annotated navigation topology. Our architecture distills the scene into a 2D occupancy map, extracts its topological layout, and overlays a lightweight semantic layer via intelligent keyframe and semantic selection. We demonstrate the versatility of this structured spatial knowledge through critical downstream Human-AI interaction tasks: (1) an intent-driven Semantic Search engine that actively infers categorical alternatives and zones when exact matches fail; (2) a one-shot Semantic Localizer achieving a 1.04 m top-5 mean translation error; (3) a Zone Classification module that segments the walkable floor plan into high-level semantic regions; and (4) a Visually-Grounded Instruction Generator that synthesizes optimal paths into egocentric, landmark-rich natural language routing. In multi-criteria LLM evaluations, GIST outperforms sequence-based instruction generation baselines. Finally, an in-situ formative evaluation (N=5) yields an 80% navigation success rate relying solely on verbal cues, validating the system's capacity for universal design.

Problem

• spatial grounding
• multimodal knowledge extraction
• cluttered environments
• semantic topology
• embodied AI

Innovation

• spatial grounding
• multimodal knowledge extraction
• semantic topology
• Vision-Language Models
• egocentric navigation