Chance and Mass Interpretations of Probabilities in Markov Decision Processes (Extended Version)

2025-06-12

AI Summary

This paper addresses the long-standing semantic divergence in Markov decision processes (MDPs) between the “chance interpretation” (viewing MDPs as state transformers) and the “mass interpretation” (viewing MDPs as distribution transformers). We propose Chance-Mass (CM) classifiers—the first unified semantic framework systematically integrating four probabilistic interpretations, including two novel ones. Methodologically, we rigorously formalize the logical relationships among these interpretations using formal semantics and probability theory; prove that reachability is PSPACE-hard under both new interpretations; and design and validate two exact algorithms for quantitative verification. Our main contributions are: (i) the first formal unification of multiple probabilistic interpretations of MDPs; (ii) an expansion of the boundaries of semantic modeling for probabilistic systems; and (iii) complexity benchmarks and computational tools enabling verification across diverse interpretations.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Machine Learning: Probabilistic Circuits and Graphical Models

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Markov decision processes (MDPs) are a popular model for decision-making in the presence of uncertainty. The conventional view of MDPs in verification treats them as state transformers with probabilities defined over sequences of states and with schedulers making random choices. An alternative view, especially well-suited for modeling dynamical systems, defines MDPs as distribution transformers with schedulers distributing probability masses. Our main contribution is a unified semantical framework that accommodates these two views and two new ones. These four semantics of MDPs arise naturally through identifying different sources of randomness in an MDP (namely schedulers, configurations, and transitions) and providing different ways of interpreting these probabilities (called the chance and mass interpretations). These semantics are systematically unified through a mathematical construct called chance-mass (CM) classifier. As another main contribution, we study a reachability problem in each of the two new semantics, demonstrating their hardness and providing two algorithms for solving them.

Problem

• Unified framework for chance and mass interpretations in MDPs
• Identify sources of randomness in Markov decision processes
• Study reachability problem in new MDP semantics

Innovation

• Unified semantical framework for MDPs
• Chance-mass classifier integrates interpretations
• Algorithms for new reachability semantics