Sample-Efficient Learning of Probabilistic Causes for Reachability in Markov Decision Processes with Probabilistic Guarantees

2026-06-28

AI Summary

This work addresses the challenge of efficiently identifying causal states—referred to as PR causes—that increase the probability of reaching a target state in Markov decision processes (MDPs) with unknown transition dynamics. The authors propose a restart-based MDP reconstruction method that reduces PR causality verification to two conditional reachability queries, eliminating the need for prior knowledge of the original MDP’s reachability probabilities. By integrating two-sided value iteration with statistical learning, the approach achieves, for the first time, sample-efficient learning of PR causes in unknown MDPs with rigorous probabilistic guarantees and provides theoretical bounds on sample complexity. Empirical evaluations on two benchmark tasks demonstrate that the algorithm reliably and rapidly identifies PR causal states, significantly outperforming conventional model-based methods that require full knowledge of the underlying MDP.

Abstract

Probabilistic model checking for Markov decision processes (MDPs) provides quantitative guarantees, but often offers limited insight into why undesired outcomes occur. Probability-raising (PR) causality addresses this by identifying states whose visitation increases the probability of reaching designated states. Existing PR-cause identification methods, however, use MDP modifications not well-suited for learning: the gap between conditional and unconditional reachability probabilities can be hard to detect from transition samples, and construction requires reachability probabilities of the MDP, which are unavailable when transition probabilities are unknown. We study unknown MDPs and propose a learning approach with probabilistic guarantees for PR-cause identification. Our key ingredient is a restart-based MDP modification that reduces PR-cause checking to two conditional reachability queries without using reachability values of the original MDP. We prove correctness, establish sample-complexity bounds, and develop an anytime learning-and-checking algorithm based on two-sided value iteration that progressively classifies states as causal, non-causal, or undecided. Experiments on two benchmarks demonstrate reliable and fast identification of PR causes.

Problem

• probabilistic causality
• Markov decision processes
• sample-efficient learning
• reachability
• probabilistic guarantees

Innovation

• probabilistic causality
• Markov decision processes
• sample-efficient learning
• restart-based modification
• conditional reachability