SARO: Space-Aware Robot System for Terrain Crossing via Vision-Language Model

2024-07-23

AI Summary

To address the challenges of autonomous navigation and poor generalization for quadrupedal robots in complex 3D terrain, this paper proposes a hierarchical closed-loop navigation framework integrating Vision-Language Models (VLMs). Methodologically: (1) VLMs enable high-level semantic understanding and generalizable task decomposition; (2) a closed-loop subtask execution mechanism ensures robust and reliable policy execution; (3) a low-level controller based on Probabilistic Annealing Selection (PAS)—a reinforcement learning technique—is introduced to enhance locomotion robustness and cross-terrain adaptability. Experiments demonstrate that the system achieves high-precision goal-reaching across diverse indoor and outdoor 3D environments, significantly outperforming baseline methods. It exhibits strong scene generalization capability and is amenable to end-to-end deployment on real-world robotic platforms.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Safety and Robustness
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The application of vision-language models (VLMs) has achieved impressive success in various robotics tasks. However, there are few explorations for these foundation models used in quadruped robot navigation through terrains in 3D environments. In this work, we introduce SARO (Space Aware Robot System for Terrain Crossing), an innovative system composed of a high-level reasoning module, a closed-loop sub-task execution module, and a low-level control policy. It enables the robot to navigate across 3D terrains and reach the goal position. For high-level reasoning and execution, we propose a novel algorithmic system taking advantage of a VLM, with a design of task decomposition and a closed-loop sub-task execution mechanism. For low-level locomotion control, we utilize the Probability Annealing Selection (PAS) method to effectively train a control policy by reinforcement learning. Numerous experiments show that our whole system can accurately and robustly navigate across several 3D terrains, and its generalization ability ensures the applications in diverse indoor and outdoor scenarios and terrains. Project page: https://saro-vlm.github.io/

Problem

• Enables quadruped robots to navigate 3D terrains using vision-language models.
• Develops a system with high-level reasoning and closed-loop task execution.
• Utilizes reinforcement learning for robust low-level locomotion control.

Innovation

• Vision-language model for high-level reasoning
• Closed-loop sub-task execution mechanism
• Probability Annealing Selection for control policy