🤖 AI Summary
This work addresses risk-sensitive continuous control tasks by proposing a reinforcement learning framework that jointly models the return distribution and policy entropy. Methodologically, it unifies distributional RL and maximum-entropy RL within a single architecture: it explicitly parameterizes the quantile function to model the cumulative reward distribution and extends Soft Actor-Critic with tunable risk measures—such as Conditional Value-at-Risk (CVaR) and entropic risk—to enable flexible control over risk preference (aversion or seeking). The core contribution lies in departing from the conventional expected-return optimization paradigm, enabling end-to-end optimization of arbitrary quantile-based risk metrics. Evaluated on multiple continuous-control benchmarks and risk-sensitive domains—including obstacle avoidance and energy-efficient control—the approach consistently outperforms state-of-the-art methods, achieving superior stability and policy robustness.
📝 Abstract
Most of reinforcement learning (RL) algorithms aim at maximizing the expectation of accumulated discounted returns. Since the accumulated discounted return is a random variable, its distribution includes more information than its expectation. Meanwhile, entropy of policy indicates its diversity and it can help improve the exploration capability of algorithms. In this paper, we present a new RL algorithm named Distributional Soft Actor Critic (DSAC), combining distributional RL and maximum entropy RL together. Taking the randomness both in action and discounted return into consideration, DSAC over performs the state-of-the-art baselines with more stability in several continuous control benchmarks. Moreover, distributional information of returns can also be used to measure metrics other than expectation, such as risk-related metrics. With a fully parameterized quantile function, DSAC is easily adopted to optimize policy under different risk preferences. Our experiments demonstrate that with distribution modeling in RL the agent performs better both for risk-averse and risk-seeking control tasks.
