On the Fundamental Limitations of Dual Static CVaR Decompositions in Markov Decision Processes

2025-07-18

AI Summary

This paper identifies the fundamental reason why static Conditional Value-at-Risk (CVaR) optimal policies in Markov decision processes (MDPs) cannot be obtained via standard dual CVaR dynamic programming: implicit inconsistency in risk allocation across states under conventional dual decomposition induces a systematic gap in policy evaluation, quantifiable explicitly. Method: We introduce the novel concept of *risk allocation consistency constraints* and rigorously prove that no static policy can be globally optimal across all initial CVaR risk levels. Our analysis integrates dynamic programming, dual CVaR decomposition, and constrained optimization. Contribution/Results: We formally characterize the intrinsic failure mechanism of existing methods and, for the first time, establish precise existence boundaries for uniformly optimal static CVaR policies. This provides both theoretical foundations and essential feasibility constraints for risk-aware sequential decision-making under distributional robustness.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Recent work has shown that dynamic programming (DP) methods for finding static CVaR-optimal policies in Markov Decision Processes (MDPs) can fail when based on the dual formulation, yet the root cause for the failure has remained unclear. We expand on these findings by shifting focus from policy optimization to the seemingly simpler task of policy evaluation. We show that evaluating the static CVaR of a given policy can be framed as two distinct minimization problems. For their solutions to match, a set of ``risk-assignment consistency constraints'' must be satisfied, and we demonstrate that the intersection of the constraints being empty is the source of previously observed evaluation errors. Quantifying the evaluation error as the CVaR evaluation gap, we then demonstrate that the issues observed when optimizing over the dual-based CVaR DP are explained by the returned policy having a non-zero CVaR evaluation gap. We then leverage our proposed risk-assignment perspective to prove that the search for a single, uniformly optimal policy via on the dual CVaR decomposition is fundamentally limited, identifying an MDP where no single policy can be optimal across all initial risk levels.

Problem

• Identify root cause of dual CVaR DP evaluation errors
• Prove risk-assignment constraints determine CVaR evaluation gap
• Show dual CVaR decomposition cannot yield uniformly optimal policies

Innovation

• Framing static CVaR evaluation as dual minimization problems
• Identifying risk-assignment consistency constraints as error source
• Proving dual CVaR decomposition's fundamental optimization limits