FastDSAC: Unlocking the Potential of Maximum Entropy RL in High-Dimensional Humanoid Control

2026-03-12

AI Summary

This work addresses the challenges of inefficient exploration and training instability in maximum-entropy reinforcement learning for high-dimensional humanoid control, which stem from the curse of dimensionality. To overcome these issues, the authors propose a Dimension-wise Entropy Modulation (DEM) mechanism that dynamically allocates exploration resources across individual action dimensions. This approach is further integrated with a continuous distributional critic to mitigate value overestimation in high-dimensional action spaces. The resulting method significantly enhances both the efficiency and stability of stochastic policy learning. Empirical evaluations on the HumanoidBench benchmark demonstrate that the proposed approach outperforms existing deterministic methods, achieving performance improvements of 180% and 400% on the Basketball and Balance Hard tasks, respectively.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Reinforcement Learning
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Scaling Maximum Entropy Reinforcement Learning (RL) to high-dimensional humanoid control remains a formidable challenge, as the ``curse of dimensionality'' induces severe exploration inefficiency and training instability in expansive action spaces. Consequently, recent high-throughput paradigms have largely converged on deterministic policy gradients combined with massive parallel simulation. We challenge this compromise with FastDSAC, a framework that effectively unlocks the potential of maximum entropy stochastic policies for complex continuous control. We introduce Dimension-wise Entropy Modulation (DEM) to dynamically redistribute the exploration budget and enforce diversity, alongside a continuous distributional critic tailored to ensure value fidelity and mitigate high-dimensional value overestimation. Extensive evaluations on HumanoidBench and other continuous control tasks demonstrate that rigorously designed stochastic policies can consistently match or outperform deterministic baselines, achieving notable gains of 180\% and 400\% on the challenging \textit{Basketball} and \textit{Balance Hard} tasks.

Problem

• Maximum Entropy Reinforcement Learning
• High-Dimensional Humanoid Control
• Curse of Dimensionality
• Exploration Inefficiency
• Training Instability

Innovation

• Maximum Entropy Reinforcement Learning
• Dimension-wise Entropy Modulation
• Distributional Critic
• High-Dimensional Humanoid Control
• Stochastic Policy