Robust Humanoid Walking on Compliant and Uneven Terrain with Deep Reinforcement Learning

2024-11-22

IEEE-RAS International Conference on Humanoid Robots

AI Summary

To address the insufficient robustness of bipedal locomotion for humanoid robots on complex, deformable, and irregular terrains—such as soft soil, gravel, and sloped surfaces—this paper proposes a sim-to-real deep reinforcement learning framework relying solely on proprioceptive sensing. Methodologically, it employs the Proximal Policy Optimization (PPO) algorithm and introduces two key innovations: (1) a lightweight curriculum learning strategy based on randomized terrain generation to enhance policy generalization across diverse ground conditions; and (2) a novel tunable clock signal mechanism that enables adaptive, real-time modulation of step frequency and stance/swing phase durations, thereby improving robustness for aperiodic gaits. The approach is validated on the HRP-5P humanoid platform, achieving stable walking both indoors and outdoors over multiple challenging terrain types. It significantly outperforms baseline methods in terms of stability and adaptability. The source code is publicly available.

Technology Category

• Intelligent Robots: Motion and Path Planning
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

For the deployment of legged robots in real-world environments, it is essential to develop robust locomotion control methods for challenging terrains that may exhibit unexpected deformability and irregularity. In this paper, we explore the application of sim-to-real deep reinforcement learning (RL) for the design of bipedal locomotion controllers for humanoid robots on compliant and uneven terrains. Our key contribution is to show that a simple training curriculum for exposing the RL agent to randomized terrains in simulation can achieve robust walking on a real humanoid robot using only proprioceptive feedback. We train an end-to-end bipedal locomotion policy using the proposed approach, and show extensive real-robot demonstration on the HRP-5P humanoid over several difficult terrains inside and outside the lab environment. Further, we argue that the robustness of a bipedal walking policy can be improved if the robot is allowed to exhibit aperiodic motion with variable stepping frequency. We propose a new control policy to enable modification of the observed clock signal, leading to adaptive gait frequencies depending on the terrain and command velocity. Through simulation experiments, we show the effectiveness of this policy specifically for walking over challenging terrains by controlling swing and stance durations. The code for training and evaluation is available online. https://github.com/rohanpsingh/LearningHumanoidWalking

Problem

• Develop robust locomotion for humanoid robots on uneven terrain
• Use deep reinforcement learning for sim-to-real control
• Improve walking robustness with adaptive gait frequencies

Innovation

• Sim-to-real deep reinforcement learning for locomotion
• Training curriculum with randomized terrains in simulation
• Adaptive gait frequencies via modified clock signal