Efficient Learning for Entropy-regularized Markov Decision Processes via Multilevel Monte Carlo

2025-03-27

AI Summary

This paper addresses entropy-regularized Markov decision processes (MDPs) in Polish spaces—i.e., large or continuous state-action spaces—where conventional methods suffer from exponential sample complexity dependence on dimensionality. Method: We propose the first learning algorithm integrating multilevel Monte Carlo (MLMC) with fixed-point iteration. It employs an unbiased randomized MLMC estimator to approximate the entropy-regularized Bellman operator and couples it with stochastic approximation for policy updates. Contribution/Results: We establish a dimension-free polynomial sample complexity bound, breaking the curse of dimensionality inherent in standard Monte Carlo–based approaches. This work introduces MLMC to entropy-regularized MDPs for the first time and develops a novel bias–variance co-analysis framework. Numerical experiments on high-dimensional continuous control tasks demonstrate that our algorithm significantly outperforms standard Monte Carlo methods and maintains robust performance as state-action space dimensionality increases.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Online Learning & Bandits
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Large-scale security measurements

Abstract

Designing efficient learning algorithms with complexity guarantees for Markov decision processes (MDPs) with large or continuous state and action spaces remains a fundamental challenge. We address this challenge for entropy-regularized MDPs with Polish state and action spaces, assuming access to a generative model of the environment. We propose a novel family of multilevel Monte Carlo (MLMC) algorithms that integrate fixed-point iteration with MLMC techniques and a generic stochastic approximation of the Bellman operator. We quantify the precise impact of the chosen approximate Bellman operator on the accuracy of the resulting MLMC estimator. Leveraging this error analysis, we show that using a biased plain MC estimate for the Bellman operator results in quasi-polynomial sample complexity, whereas an unbiased randomized multilevel approximation of the Bellman operator achieves polynomial sample complexity in expectation. Notably, these complexity bounds are independent of the dimensions or cardinalities of the state and action spaces, distinguishing our approach from existing algorithms whose complexities scale with the sizes of these spaces. We validate these theoretical performance guarantees through numerical experiments.

Problem

• Efficient learning for entropy-regularized MDPs with large state-action spaces
• Designing algorithms with quasi-polynomial or polynomial sample complexity
• Achieving complexity bounds independent of state-action space dimensions

Innovation

• Multilevel Monte Carlo for entropy-regularized MDPs
• Biased and unbiased Bellman operator approximations
• Sample complexity independent of state-action dimensions