Multimodal LLM Guided Exploration and Active Mapping using Fisher Information

2024-10-22

AI Summary

Existing active mapping approaches fail to effectively integrate multimodal large language models (MLLMs) and neglect the critical impact of localization uncertainty on embodied agents. This paper proposes the first semantic-driven active mapping framework tailored for embodied agents: an MLLM serves as a zero-shot semantic planner enabling long-horizon, task-oriented exploration decisions; concurrently, a short-horizon action optimization mechanism—grounded in 3D Gaussian Splatting (3DGS) representation and Fisher information theory—explicitly accounts for localization uncertainty, jointly maximizing environmental information gain while minimizing pose estimation error. Evaluated on the Gibson and Habitat-Matterport benchmarks, our method achieves state-of-the-art performance, improving mapping completeness by +12.7% and enhancing localization robustness with a 23.4% reduction in pose estimation error.

Technology Category

• Intelligent Robots: Localization, Mapping, and Navigation
• Planning, Routing, and Scheduling: Planning with Language Models
• Multiagent Systems: Multiagent Planning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We present an active mapping system that could plan for long-horizon exploration goals and short-term actions with a 3D Gaussian Splatting (3DGS) representation. Existing methods either did not take advantage of recent developments in multimodal Large Language Models (LLM) or did not consider challenges in localization uncertainty, which is critical in embodied agents. We propose employing multimodal LLMs for long-horizon planning in conjunction with detailed motion planning using our information-based algorithm. By leveraging high-quality view synthesis from our 3DGS representation, our method employs a multimodal LLM as a zero-shot planner for long-horizon exploration goals from the semantic perspective. We also introduce an uncertainty-aware path proposal and selection algorithm that balances the dual objectives of maximizing the information gain for the environment while minimizing the cost of localization errors. Experiments conducted on the Gibson and Habitat-Matterport 3D datasets demonstrate state-of-the-art results of the proposed method.

Problem

• Active mapping with long-horizon exploration and short-term planning
• Addressing localization uncertainty in embodied agent navigation
• Leveraging multimodal LLMs for semantic goal-oriented exploration

Innovation

• Multimodal LLM for long-horizon semantic exploration planning
• 3D Gaussian Splatting representation enabling high-quality view synthesis
• Uncertainty-aware path algorithm balancing information gain and localization cost