HierKick: Hierarchical Reinforcement Learning for Vision-Guided Soccer Robot Control

2026-03-01

AI Summary

This work addresses the challenge of multi-timescale decision-making for soccer robots in complex dynamic environments by proposing a dual-frequency hierarchical reinforcement learning architecture. The high-level policy operates at 5 Hz, integrating real-time visual detection from YOLOv8 with a coach model to select tactical tasks, while the low-level controller runs at 50 Hz, driving pretrained motion modules to precisely execute four sequential actions: approaching, aligning, dribbling, and kicking. This modular hierarchical paradigm facilitates skill reuse and task extensibility, significantly enhancing system adaptability. Experimental results demonstrate task success rates of 95.2%, 89.8%, and 80% on the IsaacGym and MuJoCo simulators and real-world robotic platforms, respectively, offering an effective new approach to multiscale control in intelligent robotics.

Technology Category

• Intelligent Robots: Multi-Robot Systems
• Multiagent Systems: Multiagent Learning
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Controlling soccer robots involves multi-time-scale decision-making, which requires balancing long-term tactical planning and short-term motion execution. Traditional end-to-end reinforcement learning (RL) methods face challenges in complex dynamic environments. This paper proposes HierKick, a vision-guided soccer robot control framework based on dual-frequency hierarchical RL. The framework adopts a hierarchical control architecture featuring a 5 Hz high-level policy that integrates YOLOv8 for real-time detection and selects tasks via a coach model, and a pre-trained 50 Hz low-level controller for precise joint control. Through this architecture, the framework achieves the four steps of approaching, aligning, dribbling, and kicking. Experimental results show that the success rates of this framework are 95.2\% in IsaacGym, 89.8\% in Mujoco, and 80\% in the real world. HierKick provides an effective hierarchical paradigm for robot control in complex environments, extendable to multi-time-scale tasks, with its modular design and skill reuse offering a new path for intelligent robot control.

Problem

• hierarchical reinforcement learning
• soccer robot control
• multi-time-scale decision-making
• vision-guided control
• complex dynamic environments

Innovation

• Hierarchical Reinforcement Learning
• Vision-Guided Control
• Dual-Frequency Architecture
• Modular Robot Control
• Multi-Time-Scale Decision Making