Sound Value Iteration for Simple Stochastic Games

2025-09-17

AI Summary

Existing sound value iteration (SVI) methods fail to handle Markov decision processes (MDPs) or stochastic games (SGs) containing end components, lacking theoretical guarantees on convergence and precision. Method: We propose the first exact value iteration framework scalable to SGs and MDPs with end components. Our approach combines graph-theoretic decomposition to identify end components, rigorous interval-based estimation, and a novel end-component resolution mechanism supported by formal convergence analysis. We further introduce multi-level pruning and adaptive precision control to accelerate convergence and tighten upper and lower bounds. Results: Experiments demonstrate that our method converges faster and achieves higher precision than state-of-the-art SVI on models with probabilistic cycles. It constitutes the first value iteration technique for probabilistic systems that simultaneously provides strong theoretical soundness—guaranteeing correctness, convergence, and error bounds—and practical scalability, thereby enabling rigorous formal verification of complex stochastic systems.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Multiagent Systems: Adversarial Agents

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Algorithmic analysis of Markov decision processes (MDP) and stochastic games (SG) in practice relies on value-iteration (VI) algorithms. Since basic VI does not provide guarantees on the precision of the result, variants of VI have been proposed that offer such guarantees. In particular, sound value iteration (SVI) not only provides precise lower and upper bounds on the result, but also converges faster in the presence of probabilistic cycles. Unfortunately, it is neither applicable to SG, nor to MDP with end components. In this paper, we extend SVI and cover both cases. The technical challenge consists mainly in proper treatment of end components, which require different handling than in the literature. Moreover, we provide several optimizations of SVI. Finally, we evaluate our prototype implementation experimentally to demonstrate its potential on systems with probabilistic cycles.

Problem

• Extends sound value iteration to stochastic games
• Handles end components in Markov decision processes
• Provides optimized algorithms for probabilistic cycles

Innovation

• Extends sound value iteration to stochastic games
• Handles end components with novel treatment approach
• Optimizes algorithm for systems with probabilistic cycles