Lyapunov-Based Sample Complexity Analysis for Weakly-Coupled MDPs

2026-06-12

AI Summary

This work addresses the exponential explosion of the state-action space with the number of arms $N$ in weakly coupled Markov decision processes (WCMDPs) and restless bandits (RBs) under the average-reward setting. It presents the first finite-sample PAC guarantee applicable to fully heterogeneous WCMDPs. By exploiting the weak coupling structure, the approach integrates generative model estimation, plug-in learning, and efficient planning within a novel analytical framework based on an explicit Lyapunov function—replacing traditional methods reliant on hard-to-control bias functions—and introduces refined perturbation analyses via drift transformations and linear programming relaxations. The resulting algorithm achieves polynomial computational and sample complexity in $N$, attaining an $O(1/\sqrt{N})$ optimality gap for heterogeneous WCMDPs and further tightening this gap in the homogeneous RB case.

Abstract

We study the sample complexity of learning in average-reward weakly-coupled Markov decision processes (WCMDPs) and Restless Bandits (RBs) under a generative model. Naive reduction to a tabular MDP leads to high complexity bounds as the state-action space is exponentially large in the number of arms $N$. By exploiting the weakly coupled structure, we show that near-optimal policies can be learned with sample and computational complexities that are polynomial in $N$. Specifically, we analyze the plug-in approach, which applies an efficient planning algorithm to an empirical model estimated from data. For fully heterogeneous WCMDPs, we establish the first finite-sample PAC guarantee with polynomial complexity and an $O(1/\sqrt{N})$ optimality gap. For homogeneous RBs, we further prove that a smaller optimality gap is achievable under mild structural assumptions. A primary technical contribution of our work is a novel Lyapunov-based analysis framework. Unlike classical approaches that rely on the difficult-to-control bias function, our framework uses an explicitly constructed Lyapunov function along with a drift transfer technique between the true and empirical models. A key step of independent interest in our framework is a fine-grained perturbation analysis for the underlying linear programming (LP) relaxation, which provides a general tool for analyzing LP-based policies and weakly-coupled systems.

Problem

• sample complexity
• weakly-coupled MDPs
• restless bandits
• average-reward
• Lyapunov analysis

Innovation

• Lyapunov-based analysis
• weakly-coupled MDPs
• sample complexity
• restless bandits
• linear programming relaxation