Dynamic Legged Ball Manipulation on Rugged Terrains with Hierarchical Reinforcement Learning

2025-04-21

AI Summary

This work addresses two key challenges in dynamic ball dribbling for quadrupedal robots on rough terrain: (1) strong coupling among locomotion modalities and (2) inefficient training due to sparse rewards. To this end, we propose a hierarchical reinforcement learning framework: a high-level policy adaptively schedules pre-trained low-level skills (e.g., walking, turning, kicking), while a low-level controller executes embodied perception-driven fine-grained control. We further introduce Dynamic Skill-Focused Policy Optimization (DSFPO), which suppresses gradient interference from inactive skills during training, thereby improving convergence and generalization of critical skills. Our approach integrates proprioceptive sensor fusion, sim-to-real transfer, and skill library reuse. Evaluated on both simulation and real-world A1 robots, it significantly outperforms baseline methods, achieving high-speed, robust dynamic dribbling across complex terrains—including gravel, inclined slopes, and stair-like steps.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Reinforcement Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Advancing the dynamic loco-manipulation capabilities of quadruped robots in complex terrains is crucial for performing diverse tasks. Specifically, dynamic ball manipulation in rugged environments presents two key challenges. The first is coordinating distinct motion modalities to integrate terrain traversal and ball control seamlessly. The second is overcoming sparse rewards in end-to-end deep reinforcement learning, which impedes efficient policy convergence. To address these challenges, we propose a hierarchical reinforcement learning framework. A high-level policy, informed by proprioceptive data and ball position, adaptively switches between pre-trained low-level skills such as ball dribbling and rough terrain navigation. We further propose Dynamic Skill-Focused Policy Optimization to suppress gradients from inactive skills and enhance critical skill learning. Both simulation and real-world experiments validate that our methods outperform baseline approaches in dynamic ball manipulation across rugged terrains, highlighting its effectiveness in challenging environments. Videos are on our website: dribble-hrl.github.io.

Problem

• Coordinate motion modalities for terrain traversal and ball control
• Overcome sparse rewards in end-to-end reinforcement learning
• Enhance dynamic ball manipulation on rugged terrains

Innovation

• Hierarchical reinforcement learning for dynamic ball manipulation
• Dynamic Skill-Focused Policy Optimization enhances critical skills
• High-level policy adaptively switches pre-trained low-level skills