Imitating and Finetuning Model Predictive Control for Robust and Symmetric Quadrupedal Locomotion

2023-11-01

IEEE Robotics and Automation Letters

AI Summary

To address the challenge of achieving robust, symmetric, periodic, and energy-efficient gaits for quadrupedal robots on unstructured terrains—including uneven ground, slippery surfaces, and moving conveyor belts—this paper proposes an “Imitation–Fine-tuning” collaborative control framework (IFM). IFM innovatively employs a model predictive controller (MPC) combining differential dynamic programming with Raibert-inspired heuristics as an expert policy, which is first rendered learnable via behavioral cloning and subsequently refined safely using low-exploration-depth PPO or SAC reinforcement learning. Comprehensive simulation and hardware experiments demonstrate that IFM significantly improves gait stability and left–right symmetry, generates more periodic and energy-efficient locomotion patterns, and eliminates the need for intricate reward engineering. The framework achieves a favorable trade-off among safety, environmental adaptability, and deployment efficiency.

Abstract

Control of legged robots is a challenging problem that has been investigated by different approaches, such as model-based control and learning algorithms. This work proposes a novel Imitating and Finetuning Model Predictive Control (IFM) framework to take the strengths of both approaches. Our framework first develops a conventional model predictive controller (MPC) using Differential Dynamic Programming and Raibert heuristic, which serves as an expert policy. Then we train a clone of the MPC using imitation learning to make the controller learnable. Finally, we leverage deep reinforcement learning with limited exploration for further finetuning the policy on more challenging terrains. By conducting comprehensive simulation and hardware experiments, we demonstrate that the proposed IFM framework can significantly improve the performance of the given MPC controller on rough, slippery, and conveyor terrains that require careful coordination of footsteps. We also showcase that IFM can efficiently produce more symmetric, periodic, and energy-efficient gaits compared to Vanilla RL with a minimal burden of reward shaping.

Problem

• Improving quadruped robot locomotion on challenging terrains
• Combining model-based control with learning algorithms
• Enhancing gait symmetry and energy efficiency

Innovation

• Combining model predictive control with imitation learning
• Finetuning policy via deep reinforcement learning
• Enhancing robustness on challenging terrains efficiently