Ergodic Unobservable MDPs: Decidability of Approximation

2024-05-21

arXiv.org

AI Summary

This paper investigates the existence of uniform values in blind stochastic games—stochastic games with unobservable states—and focuses on their subclass: ergodic blind stochastic games. For the corresponding unobservable Markov decision processes (UMDPs), we introduce the novel notion of “ergodic UMDPs” and establish an asymptotic behavior analysis framework grounded in matrix products and the ergodicity of associated Markov chains. Our main contributions are threefold: (1) We prove that the long-run average payoff’s ε-approximation is decidable for ergodic UMDPs and provide an effective algorithm; (2) we show that computing the exact value remains undecidable, thereby precisely delineating the boundary of computability; (3) we fill a critical gap in UMDP theory by providing the first decidability analysis, establishing a new theoretical benchmark for partially observable reinforcement learning.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Multiagent Systems: Multiagent Systems under Uncertainty
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Unobservable Markov decision processes (UMDPs) serve as a prominent mathematical framework for modeling sequential decision-making problems. A key aspect in computational analysis is the consideration of decidability, which concerns the existence of algorithms. In general, the computation of the exact and approximated values is undecidable for UMDPs with the long-run average objective. Building on matrix product theory and ergodic properties, we introduce a novel subclass of UMDPs, termed ergodic UMDPs. Our main result demonstrates that approximating the value within this subclass is decidable. However, we show that the exact problem remains undecidable. Finally, we discuss the primary challenges of extending these results to partially observable Markov decision processes.

Problem

• Existence of uniform value in ergodic blind stochastic games
• Decidability of approximating uniform value in ergodic games
• Independence of uniform value from initial belief state

Innovation

• Introduces ergodic blind stochastic games subclass
• Proves uniform value existence with ergodicity condition
• Provides algorithm to approximate uniform value