Robust Quadruped Locomotion via Evolutionary Reinforcement Learning

2026-04-08

AI Summary

This work addresses the challenge of poor generalization of locomotion policies trained in simulation to unseen complex terrains, such as rough ground, for quadrupedal robots. To tackle this issue, the authors propose an evolutionary reinforcement learning framework that integrates gradient-based optimization with population-based exploration. Specifically, they combine the Cross-Entropy Method (CEM) with TD3 and DDPG to develop CEM-TD3 and CEM-DDPG algorithms. Policies trained on flat terrain using these methods demonstrate significantly enhanced transferability and robustness when deployed on previously unencountered rugged environments. Experimental results show that CEM-TD3 achieves the best generalization performance, attaining an average reward of 19,574.33—substantially outperforming conventional deep reinforcement learning approaches—and effectively mitigates overfitting while improving environmental adaptability.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Machine Learning: Evolutionary Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Deep reinforcement learning has recently achieved strong results in quadrupedal locomotion, yet policies trained in simulation often fail to transfer when the environment changes. Evolutionary reinforcement learning aims to address this limitation by combining gradient-based policy optimisation with population-driven exploration. This work evaluates four methods on a simulated walking task: DDPG, TD3, and two Cross-Entropy-based variants CEM-DDPG and CEM-TD3. All agents are trained on flat terrain and later tested both on this domain and on a rough terrain not encountered during training. TD3 performs best among the standard deep RL baselines on flat ground with a mean reward of 5927.26, while CEM-TD3 achieves the highest rewards overall during training and evaluation 17611.41. Under the rough-terrain transfer test, performance of the deep RL methods drops sharply. DDPG achieves -1016.32 and TD3 achieves -99.73, whereas the evolutionary variants retain much of their capability. CEM-TD3 records the strongest transfer performance with a mean reward of 19574.33. These findings suggest that incorporating evolutionary search can reduce overfitting and improve policy robustness in locomotion tasks, particularly when deployment conditions differ from those seen during training.

Problem

• quadruped locomotion
• sim-to-real transfer
• policy robustness
• domain generalization
• reinforcement learning

Innovation

• Evolutionary Reinforcement Learning
• Quadruped Locomotion
• Policy Robustness
• Domain Transfer
• CEM-TD3