FastDSAC: Enhancing Policy Plasticity via Constrained Exploration for Scalable Humanoid Locomotion

2026-06-30

AI Summary

This work addresses the instability in value estimation and degradation of policy plasticity commonly observed in high-throughput sampling scenarios with frequent data updates. To mitigate these issues, the authors propose FastDSAC, an algorithm built upon a distributed Actor-Critic framework that models the policy using a truncated Gaussian distribution to simultaneously respect action constraints and preserve exploratory stochasticity. The method incorporates an adaptive variance modulation mechanism to enhance the accuracy of value estimation and employs implicit regularization to maintain the adaptability of the policy network. Experimental results demonstrate that FastDSAC achieves more stable training dynamics, faster convergence, and superior asymptotic performance compared to existing approaches on both the MuJoCo Playground and HumanoidBench benchmarks.

Abstract

Scalable reinforcement learning has popularized high-throughput sampling architectures, which significantly compresses the training time for off-policy methods in robotic locomotion. However, the rapid increase of data volume and update frequency undermines the stability of value-based methods and diminishes the plasticity of policy networks. To address these challenges, this work presents FastDSAC, a fast and high-performance variant of the Distributional Actor-Critic algorithm designed for parallel sampling scenarios. Specifically, we introduce a truncated Gaussian distribution to approximate the learned policy, which effectively excludes out-of-distribution actions that strain target value estimation while keeping necessary stochasticity for exploration. The proposed action constraint functions as an implicit regularization, which counteracts the plasticity loss typically caused by aggressive gradient updates. This preservation of network adaptability enhances sample efficiency, particularly in scenarios with a high update-to-data ratio, and accelerates the early training process. In contrast to prior fast reinforcement learning approaches that rely on discrete value distributions, our method utilizes a continuous Gaussian representation equipped with adaptive variance regulation, which improves value estimation accuracy by sampling confident and informative transitions. Extensive experiments on MuJoCo Playground and HumanoidBench demonstrate that FastDSAC not only stabilizes the overall training process but also achieves superior asymptotic performance and faster convergence compared to state-of-the-art baselines.

Problem

• scalable reinforcement learning
• policy plasticity
• humanoid locomotion
• value estimation stability
• high-throughput sampling

Innovation

• Constrained Exploration
• Policy Plasticity
• Distributional Actor-Critic
• Truncated Gaussian Policy
• Scalable Reinforcement Learning