Confidence Sequences for Online Statistical Model Checking of Markov Decision Processes

2026-06-24

AI Summary

This work addresses the challenge of verifying Markov decision processes (MDPs) with unknown transition probabilities that exhibit both nondeterminism and probabilistic uncertainty. To tackle this problem, the authors propose an online statistical model checking method grounded in confidence sequences. By integrating dynamic sampling, statistical hypothesis testing, and a novel online confidence sequence construction, the approach effectively mitigates the conservativeness and inefficiency inherent in traditional union-bound techniques. The resulting specialized verification tool maintains rigorous reliability guarantees while achieving a dramatic reduction in sample complexity—requiring on average approximately 50 times fewer samples than the current state-of-the-art methods—thereby substantially enhancing both efficiency and practical applicability.

Abstract

Markov decision processes (MDPs) are a classic model of decision making under uncertainty, exhibiting both non-deterministic choice as well as probabilistic uncertainty. Traditionally, exact knowledge of the underlying probabilities is assumed. However, this often is unrealistic, e.g.\ when modelling cyber-physical systems or biological processes. Here, statistical methods provide a way towards obtaining meaningful guarantees. The classical approach is to gather samples in the MDP, use these to draw statistical conclusions about the transition probabilities, and from there obtain bounds on the true value; then, if these bounds are too broad, repeat. However, existing implementations of this approach are either subtly incorrect or sub-optimal, and quite often both. We present several \emph{confidence sequences}, which are specifically designed for such \enquote{online} settings, implement all of them in an efficient tool, and show their practical applicability. In particular, we show that they outperform classical \enquote{union-bound} style approaches, and overall our implementation requires 50x less samples on average than previous state of the art.

Problem

• Markov Decision Processes
• Statistical Model Checking
• Confidence Sequences
• Online Verification
• Transition Probabilities

Innovation

• confidence sequences
• online statistical model checking
• Markov decision processes
• sample efficiency
• probabilistic verification