Mapping the Unseen: Unified Promptable Panoptic Mapping with Dynamic Labeling using Foundation Models

2024-05-03

arXiv.org

AI Summary

Traditional panoramic semantic mapping is constrained by predefined categories and struggles to recognize unknown objects. To address this, we propose the first promptable unified panoramic mapping framework. Our method integrates natural language prompts with multimodal foundation models (CLIP and SAM) to construct a prompt-driven dynamic labeling module, enabling real-time open-vocabulary semantic parsing. By jointly leveraging 3D reconstruction and instance segmentation, the framework achieves end-to-end promptable panoramic mapping. Extensive evaluation on both real-world and synthetic datasets demonstrates significant improvements in unknown-object segmentation accuracy and semantic labeling fidelity, while enabling natural-language-guided interactive map construction. Ablation studies further validate that foundation-model-based dynamic labeling substantially outperforms conventional fixed-label paradigms. This work establishes a new paradigm for flexible, scalable, and user-controllable semantic mapping beyond closed-set assumptions.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In the field of robotics and computer vision, efficient and accurate semantic mapping remains a significant challenge due to the growing demand for intelligent machines that can comprehend and interact with complex environments. Conventional panoptic mapping methods, however, are limited by predefined semantic classes, thus making them ineffective for handling novel or unforeseen objects. In response to this limitation, we introduce the Unified Promptable Panoptic Mapping (UPPM) method. UPPM utilizes recent advances in foundation models to enable real-time, on-demand label generation using natural language prompts. By incorporating a dynamic labeling strategy into traditional panoptic mapping techniques, UPPM provides significant improvements in adaptability and versatility while maintaining high performance levels in map reconstruction. We demonstrate our approach on real-world and simulated datasets. Results show that UPPM can accurately reconstruct scenes and segment objects while generating rich semantic labels through natural language interactions. A series of ablation experiments validated the advantages of foundation model-based labeling over fixed label sets.

Problem

• Dynamic labeling for novel objects in panoptic mapping
• Overcoming fixed label limitations in semantic mapping
• Leveraging foundation models for training-free semantic enrichment

Innovation

• Leverages foundation models for dynamic labeling
• Unified Promptable Panoptic Mapping without training
• Custom NMS improves completion ratio significantly