Mean-Variance Optimization and Algorithm for Finite-Horizon Markov Decision Processes

2025-07-29

AI Summary

This paper addresses the multi-period mean–variance optimization problem in finite-horizon discrete-time Markov decision processes (MDPs), a long-standing challenge due to the failure of the dynamic programming principle. To overcome limitations of conventional approaches, we propose a bilevel MDP modeling framework: by introducing pseudo-mean and pseudo-variance state variables, the nonstandard objective is reformulated as a structured optimization problem over an augmented state space. Theoretically, we establish that the inner-layer optimal value function is piecewise quadratic and concave in the pseudo-mean. Algorithmically, we design an iterative dynamic programming method that converges to a local optimum, and derive sufficient conditions for global optimality. We validate the framework on portfolio selection, queueing control, and inventory management tasks—reproducing classical results from financial engineering—and demonstrate its theoretical rigor and cross-domain applicability.

Technology Category

• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Search and Optimization: Mixed Discrete/Continuous Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Multi-period mean-variance optimization is a long-standing problem, caused by the failure of dynamic programming principle. This paper studies the mean-variance optimization in a setting of finite-horizon discrete-time Markov decision processes (MDPs), where the objective is to maximize the combined metrics of mean and variance of the accumulated rewards at terminal stage. By introducing the concepts of pseudo mean and pseudo variance, we convert the original mean-variance MDP to a bilevel MDP, where the outer is a single parameter optimization of the pseudo mean and the inner is a standard finite-horizon MDP with an augmented state space by adding an auxiliary state of accumulated rewards. We further study the properties of this bilevel MDP, including the optimality of history-dependent deterministic policies and the piecewise quadratic concavity of the inner MDPs' optimal values with respect to the pseudo mean. To efficiently solve this bilevel MDP, we propose an iterative algorithm that alternatingly updates the inner optimal policy and the outer pseudo mean. We prove that this algorithm converges to a local optimum. We also derive a sufficient condition under which our algorithm converges to the global optimum. Furthermore, we apply this approach to study the mean-variance optimization of multi-period portfolio selection problem, which shows that our approach exactly coincides with the classical result by Li and Ng (2000) in financial engineering. Our approach builds a new avenue to solve mean-variance optimization problems and has wide applicability to any problem modeled by MDPs, which is further demonstrated by examples of mean-variance optimization for queueing control and inventory management.

Problem

• Solves mean-variance optimization in finite-horizon MDPs
• Converts problem to bilevel MDP using pseudo metrics
• Proposes iterative algorithm for local/global optimum convergence

Innovation

• Introduces pseudo mean and pseudo variance
• Converts problem to bilevel MDP structure
• Proposes iterative algorithm for local optimum