Sample-Efficient Learning of Probabilistic Causes for Reachability in Markov Decision Processes with Probabilistic Guarantees
This work addresses the challenge of efficiently identifying causal states—referred to as PR causes—that increase the probability of reaching a target state in Markov decision processes (MDPs) with unknown transition dynamics. The authors propose a restart-based MDP reconstruction method that reduces PR causality verification to two conditional reachability queries, eliminating the need for prior knowledge of the original MDP’s reachability probabilities. By integrating two-sided value iteration with statistical learning, the approach achieves, for the first time, sample-efficient learning of PR causes in unknown MDPs with rigorous probabilistic guarantees and provides theoretical bounds on sample complexity. Empirical evaluations on two benchmark tasks demonstrate that the algorithm reliably and rapidly identifies PR causal states, significantly outperforming conventional model-based methods that require full knowledge of the underlying MDP.