🤖 AI Summary
This work addresses the challenge of unstable localization and degraded performance in memory-constrained edge mobile robots when concurrently deploying large language models and dense maps, primarily caused by keyframe loading latency. The study introduces indoor semantic regions—such as rooms and corridors—into dense map management and proposes a semantic-aware keyframe scheduling strategy. By integrating SLAM, vision-language models, and semantic segmentation, the method prioritizes the loading of spatially relevant map content according to semantic regions on an NVIDIA Jetson Orin Nano platform. This approach not only maintains robust localization stability but also significantly enhances large model inference efficiency, achieving a 3.3 tokens/s increase in throughput and a 21.7% reduction in latency compared to purely geometric map management, while completely eliminating memory overflow and execution stuttering.
📝 Abstract
Recent advances in large AI models (VLMs and LLMs) and joint use of the 3D dense maps, enable mobile robots to provide more powerful and interactive services grounded in rich spatial context. However, deploying both heavy AI models and dense maps on edge robots is challenging under strict memory budgets. When the memory budget is exceeded, required keyframes may not be loaded in time, which can degrade the stability of position estimation and interfering model performance. We proposes a semantic zone-based map management approach to stabilize dense-map utilization under memory constraints. We associate keyframes with semantic indoor regions (e.g., rooms and corridors) and keyframe management at the semantic zone level prioritizes spatially relevant map content while respecting memory constraints. This reduces keyframe loading and unloading frequency and memory usage. We evaluate the proposed approach in large-scale simulated indoor environments and on an NVIDIA Jetson Orin Nano under concurrent SLAM-VLM execution. With Qwen3.5:0.8b, the proposed method improves throughput by 3.3 tokens/s and reduces latency by 21.7% relative to a geometric map-management strategy. Furthermore, while the geometric strategy suffers from out-of-memory failures and stalled execution under memory pressure, the proposed method eliminates both issues, preserving localization stability and enabling robust VLM operation. These results demonstrate that the proposed approach enables efficient dense map utilization for memory constrained, AI-integrated mobile robots. Code is available at: https://github.com/huichangs/rtabmap/tree/segment