Taming Infinity one Chunk at a Time: Concisely Represented Strategies in One-Counter MDPs

2025-03-02

AI Summary

This paper investigates strategy synthesis and verification for reachability and *selective termination*—reaching a target state with the counter value exactly zero—in one-counter Markov decision processes (OC-MDPs), an infinite-state stochastic model equipped with a natural-number counter. While the decidability of selective termination had remained open for years, we resolve it by introducing two novel, compact strategy representations based on counter interval partitioning, enabling a unified compression framework for infinite MDPs. Integrating symbolic model checking, automata theory, and mathematical logic, we establish PSPACE decidability for both verification and synthesis of multiple strategy classes. Our work overturns the long-standing belief that selective termination strategy synthesis is undecidable, providing the first expressive yet computationally feasible strategy representation and analysis framework for infinite-state probabilistic systems.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Markov decision processes (MDPs) are a canonical model to reason about decision making within a stochastic environment. We study a fundamental class of infinite MDPs: one-counter MDPs (OC-MDPs). They extend finite MDPs via an associated counter taking natural values, thus inducing an infinite MDP over the set of configurations (current state and counter value). We consider two characteristic objectives: reaching a target state (state-reachability), and reaching a target state with counter value zero (selective termination). The synthesis problem for the latter is not known to be decidable and connected to major open problems in number theory. Furthermore, even seemingly simple strategies (e.g., memoryless ones) in OC-MDPs might be impossible to build in practice (due to the underlying infinite configuration space): we need finite, and preferably small, representations. To overcome these obstacles, we introduce two natural classes of concisely represented strategies based on a (possibly infinite) partition of counter values in intervals. For both classes, and both objectives, we study the verification problem (does a given strategy ensure a high enough probability for the objective?), and two synthesis problems (does there exist such a strategy?): one where the interval partition is fixed as input, and one where it is only parameterized. We develop a generic approach based on a compression of the induced infinite MDP that yields decidability in all cases, with all complexities within PSPACE.

Problem

• Deciding selective termination in one-counter MDPs.
• Representing strategies concisely in infinite MDPs.
• Verifying and synthesizing strategies for OC-MDP objectives.

Innovation

• Concisely represented strategies using interval partitions
• Compression of infinite MDPs for decidability
• PSPACE complexity for synthesis and verification