Astra: Toward General-Purpose Mobile Robots via Hierarchical Multimodal Learning

2025-06-06

AI Summary

Traditional robot navigation systems exhibit poor adaptability and modular fragmentation in complex indoor-outdoor environments. To address this, we propose a dual-model collaborative architecture: Astra-Global, a multimodal large language model (MLLM), performs semantic understanding and constructs a global topological-semantic map; Astra-Local, a multi-task neural network, enables local path planning via a 4D spatiotemporal encoder and masked ESDF loss. Our approach innovatively fuses visual, linguistic, and multi-source sensor data, incorporates self-supervised odometry estimation, and employs flow-matching-based trajectory generation. The entire system is deployed end-to-end on a custom-built robotic platform. Experiments across diverse real-world indoor scenes demonstrate that our method achieves significantly higher task success rates compared to conventional vision-based localization and rule-driven navigation baselines.

Technology Category

• Intelligent Robots: Localization, Mapping, and Navigation
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Modern robot navigation systems encounter difficulties in diverse and complex indoor environments. Traditional approaches rely on multiple modules with small models or rule-based systems and thus lack adaptability to new environments. To address this, we developed Astra, a comprehensive dual-model architecture, Astra-Global and Astra-Local, for mobile robot navigation. Astra-Global, a multimodal LLM, processes vision and language inputs to perform self and goal localization using a hybrid topological-semantic graph as the global map, and outperforms traditional visual place recognition methods. Astra-Local, a multitask network, handles local path planning and odometry estimation. Its 4D spatial-temporal encoder, trained through self-supervised learning, generates robust 4D features for downstream tasks. The planning head utilizes flow matching and a novel masked ESDF loss to minimize collision risks for generating local trajectories, and the odometry head integrates multi-sensor inputs via a transformer encoder to predict the relative pose of the robot. Deployed on real in-house mobile robots, Astra achieves high end-to-end mission success rate across diverse indoor environments.

Problem

• Improving robot navigation in complex indoor environments
• Overcoming adaptability limitations of traditional rule-based systems
• Enhancing localization and path planning with multimodal learning

Innovation

• Multimodal LLM for global navigation
• Self-supervised 4D encoder for local tasks
• Flow matching for collision-free planning