Risk-Aware General-Utility Markov Decision Processes

2026-07-10

AI Summary

This work proposes a risk-aware generalized utility Markov decision process (MDP) framework that enables flexible trade-offs between expected performance and risk aversion. The approach formulates the objective function based on state visitation frequencies and, for the first time, integrates entropic risk measures into the generalized utility MDP setting, thereby supporting risk-sensitive decision-making in multi-task scenarios. To solve this framework, the authors develop an online planning algorithm grounded in Monte Carlo tree search and establish its convergence properties. Experimental results demonstrate that the method effectively optimizes policies across diverse risk preferences in a range of tasks—including standard MDPs, maximum state-entropy exploration, imitation learning, and multi-objective MDPs—highlighting its versatility and efficacy.

Abstract

We study general-utility Markov decision processes (GUMDPs) with risk-aware objectives. In this framework, an agent aims to optimize a risk measure of the distribution of objective values, where the objective function depends on the frequency of visitation of states induced by the agent's policy. First, we motivate, propose, and formalize risk-aware GUMDPs, which enable agents and decision makers to trade off expected performance by risk aversion while benefiting from the rich set of objectives that can be cast under the framework of GUMDPs. We focus our attention on the entropic risk measure (ERM). Second, we show how we can solve risk-aware GUMDPs with ERM objectives by resorting to online planning techniques. In particular, we propose an approach based on Monte Carlo Tree Search (MCTS) to provably solve risk-aware GUMDPs up to any desired accuracy. Third, we provide a set of experimental results showcasing that our approach is successful when optimizing for a spectrum of risk-aware behaviors in the context of GUMDPs under diverse tasks (standard MDPs, maximum state entropy exploration, imitation learning, and multi-objective MDPs).

Problem

• risk-aware
• Markov decision processes
• general-utility
• entropic risk measure
• risk aversion

Innovation

• Risk-Aware MDP
• General-Utility MDP
• Entropic Risk Measure
• Monte Carlo Tree Search
• Online Planning