Non-conflicting Energy Minimization in Reinforcement Learning based Robot Control

2025-09-01

AI Summary

In robotic control via reinforcement learning, simultaneously optimizing energy efficiency and task performance remains challenging. This paper proposes a hyperparameter-free gradient projection policy optimization method: during policy gradient updates, the energy consumption gradient is orthogonally projected onto the nullspace of the task objective gradient, thereby minimizing energy expenditure without degrading task performance. To our knowledge, this is the first work to adapt multi-objective gradient projection to energy-aware control—avoiding reward shaping-induced objective conflicts and eliminating the need for hand-tuned energy penalty coefficients. The method is inherently compatible with diverse policy gradient algorithms and supports Sim2Real transfer. Evaluated on DM-Control and HumanoidBench benchmarks, it achieves up to 64% energy reduction with zero task performance degradation. Furthermore, it is successfully deployed on a Unitree GO2 quadruped robot in real-world experiments.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Search and Optimization: Learning to Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Efficient robot control often requires balancing task performance with energy expenditure. A common approach in reinforcement learning (RL) is to penalize energy use directly as part of the reward function. This requires carefully tuning weight terms to avoid undesirable trade-offs where energy minimization harms task success. In this work, we propose a hyperparameter-free gradient optimization method to minimize energy expenditure without conflicting with task performance. Inspired by recent works in multitask learning, our method applies policy gradient projection between task and energy objectives to derive policy updates that minimize energy expenditure in ways that do not impact task performance. We evaluate this technique on standard locomotion benchmarks of DM-Control and HumanoidBench and demonstrate a reduction of 64% energy usage while maintaining comparable task performance. Further, we conduct experiments on a Unitree GO2 quadruped showcasing Sim2Real transfer of energy efficient policies. Our method is easy to implement in standard RL pipelines with minimal code changes, is applicable to any policy gradient method, and offers a principled alternative to reward shaping for energy efficient control policies.

Problem

• Minimize robot energy use without affecting task performance
• Avoid trade-offs between energy savings and task success
• Develop hyperparameter-free energy optimization for reinforcement learning

Innovation

• Hyperparameter-free gradient optimization method
• Policy gradient projection between objectives
• Minimizes energy without affecting task performance