Long-horizon Locomotion and Manipulation on a Quadrupedal Robot with Large Language Models

2024-04-08

arXiv.org

AI Summary

To address the semantic gap between high-level task planning and low-level motion control in long-horizon quadrupedal robot operations, this paper proposes an LLM-RL synergistic framework. It introduces a multi-role large language model (LLM) agent—comprising semantic planners, parameter solvers, executable code generators, and dynamic replanners—tightly coupled with a reinforcement learning (RL)-based locomotion skill library and a hybrid discrete-continuous task planner, enabling end-to-end embodied reasoning and tool construction. For the first time, cross-modal autonomous navigation, multi-step tool assembly, failure detection, and proactive assistance seeking are validated both in simulation and on the physical ANYmal platform; task success rates significantly surpass those of single-skill baselines. The core contribution lies in pioneering the deep integration of multi-agent LLMs into a closed-loop embodied intelligence architecture, unifying high-level intent understanding, mid-level task decomposition, and low-level motor execution.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We present a large language model (LLM) based system to empower quadrupedal robots with problem-solving abilities for long-horizon tasks beyond short-term motions. Long-horizon tasks for quadrupeds are challenging since they require both a high-level understanding of the semantics of the problem for task planning and a broad range of locomotion and manipulation skills to interact with the environment. Our system builds a high-level reasoning layer with large language models, which generates hybrid discrete-continuous plans as robot code from task descriptions. It comprises multiple LLM agents: a semantic planner that sketches a plan, a parameter calculator that predicts arguments in the plan, a code generator that converts the plan into executable robot code, and a replanner that handles execution failures or human interventions. At the low level, we adopt reinforcement learning to train a set of motion planning and control skills to unleash the flexibility of quadrupeds for rich environment interactions. Our system is tested on long-horizon tasks that are infeasible to complete with one single skill. Simulation and real-world experiments show that it successfully figures out multi-step strategies and demonstrates non-trivial behaviors, including building tools or notifying a human for help. Demos are available on our project page: https://sites.google.com/view/long-horizon-robot.

Problem

• Enabling quadrupedal robots to perform long-horizon tasks using large language models.
• Combining high-level reasoning with low-level motion planning for complex robot behaviors.
• Addressing challenges in task planning and environment interaction for quadrupedal robots.

Innovation

• LLM-based system for quadrupedal robot tasks
• Hybrid discrete-continuous plans from task descriptions
• Reinforcement learning for motion planning skills