Energy-Efficient Arm Reaching for a Humanoid Robot via Deep Reinforcement Learning with Identified Power Models

2026-06-14

AI Summary

This work addresses the energy limitations of humanoid robots performing repeated arm-reaching tasks in unstructured environments. The authors propose an end-to-end energy-aware reinforcement learning framework that, for the first time, integrates an experimentally identified electrical power model with the Soft Actor-Critic (SAC) algorithm and introduces a mixed constellation reward mechanism to significantly reduce energy consumption while maintaining end-effector accuracy. The approach employs an incremental joint position action space and leverages Pinocchio to construct a high-fidelity dynamics simulation environment. In simulation, the method achieves a 69.9% success rate over 1,000 random targets with an average energy cost of 98.16 joules. Real-world validation demonstrates an average energy consumption of 71.5 ± 48.3 joules, with end-effector position and orientation errors of 2.64 ± 1.04 cm and 6.92 ± 1.33°, respectively, both within the training tolerance bounds.

Abstract

Humanoid robots performing in-field manipulation tasks, such as robotic apple harvesting, face severe energy constraints that directly limit the number of reaching motions that can be executed per battery charge. This paper presents an end-to-end, energy-aware reinforcement learning framework for the 7-degree-of-freedom left arm of the Unitree~G1 humanoid robot, combining a physics-based, experimentally identified electrical power model with a Soft Actor-Critic (SAC) policy trained in a Pinocchio-based rigid-body dynamics simulator. The RL policy operates on an incremental joint-position action space and is trained with a Hybrid Constellation Reward that combines a four-point end-effector constellation distance with a torque-norm energy proxy; after % $5\times10^6$ training it reaches a $69.9\%$ success rate over $1\,000$ random targets in kinematic simulation, at a mean energy of \SI{98.16}{\joule} on successful episodes. Finally, on the physical Unitree~G1, the policy is validated over three independent 10-target batches, achieving a mean energy of $71.5 \pm 48.3$\,J, an end-effector position error of $2.64 \pm 1.04$\,cm, and an orientation error of $6.92 \pm 1.33^\circ$ -- within the \SI{4}{\centi\metre}/$8.6^\circ$ training tolerance. These results constitute a first step toward energy-aware reinforcement-learning-based arm reaching for humanoid robots.

Problem

• humanoid robot
• energy-efficient
• arm reaching
• reinforcement learning
• power model

Innovation

• energy-efficient reinforcement learning
• humanoid robot arm
• power model identification
• Soft Actor-Critic
• end-effector constellation reward