Fast Computation of Conditional Probabilities in MDPs and Markov Chain Families

2026-05-12

AI Summary

This work addresses the inefficiency of traditional methods for computing optimal conditional reachability probabilities in Markov decision processes (MDPs), which suffer from complex cyclic structures introduced by reduction-based approaches. The authors propose a direct, numerically stable linear-time algorithm tailored for acyclic MDPs that circumvents these cyclic complications. By integrating this algorithm into an abstraction-refinement framework, the method enables parallelized analysis of large families of Markov chains. It combines novel techniques for conditional probability computation with advanced reachability analysis, achieving speedups of several orders of magnitude over existing approaches on benchmarks drawn from Bayesian networks, probabilistic programs, and runtime monitoring—matching the performance of standard (unconditional) reachability queries.

Abstract

Computing optimal conditional reachability probabilities in Markov decision processes (MDPs) is tractable by a reduction to reachability probabilities. Yet, this reduction yields cyclic, challenging MDPs that are often notoriously hard to solve. We present an alternative, practically efficient method to compute optimal conditional reachabilities. This new method is numerically stable, can decide the threshold problem in linear time on acyclic MDPs, and yields performance comparable to standard reachability queries. We also integrate the method in an abstraction-refinement framework to analyse millions of Markov chains at once. We demonstrate the efficacy of the new methods on benchmarks from Bayesian network analysis, probabilistic programs, and runtime monitoring and show speed-ups up to multiple orders of magnitude.

Problem

• conditional probabilities
• Markov decision processes
• reachability
• Markov chains
• probabilistic verification

Innovation

• conditional reachability
• Markov decision processes
• abstraction-refinement
• numerical stability
• linear-time algorithm