The Benefits of Being Categorical Distributional: Uncertainty-aware Regularized Exploration in Reinforcement Learning

2021-10-07

AI Summary

This paper investigates the theoretical advantages of distributional reinforcement learning (DRL) over classical RL, focusing on its implicit environmental exploration capability. Method: We provide the first rigorous decomposition of the distributional loss in categorical DRL, revealing an intrinsic, uncertainty-aware entropy regularization mechanism—spontaneously induced by the structure of the return distribution and requiring no explicit design. This adaptive regularizer transforms environmental uncertainty into enhanced reward signals for policy optimization. Unlike maximum-entropy RL, which explicitly encourages action-space diversity, this mechanism enables implicit, environment-driven exploration grounded in distributional shape. Contribution/Results: Our theoretical analysis uncovers the fundamental reason behind DRL’s superiority over classical RL. Empirical evaluation demonstrates that this implicit regularization significantly improves sample efficiency and policy robustness across diverse benchmarks.

Abstract

The remarkable empirical performance of distributional reinforcement learning (RL) has garnered increasing attention to understanding its theoretical advantages over classical RL. By decomposing the categorical distributional loss commonly employed in distributional RL, we find that the potential superiority of distributional RL can be attributed to a derived distribution-matching entropy regularization. This less-studied entropy regularization aims to capture additional knowledge of return distribution beyond only its expectation, contributing to an augmented reward signal in policy optimization. In contrast to the vanilla entropy regularization in MaxEnt RL, which explicitly encourages exploration by promoting diverse actions, the novel entropy regularization derived from categorical distributional loss implicitly updates policies to align the learned policy with (estimated) environmental uncertainty. Finally, extensive experiments substantiate the significance of this uncertainty-aware regularization from distributional RL on the empirical benefits over classical RL. Our study offers a new perspective from the exploration to explain the intrinsic benefits of adopting distributional learning in RL.

Problem

• Distributed Reinforcement Learning
• Unknown Environment Exploration
• Reward Distribution Handling

Innovation

• Distributional Reinforcement Learning
• Matching Entropy Regularization
• Enhanced Reward Signal