Efficient Strategy Synthesis for MDPs via Hierarchical Block Decomposition

2025-06-21

AI Summary

To address the poor scalability of conventional policy synthesis methods for large-scale Markov decision processes (MDPs), this paper proposes a vulnerability-driven hierarchical block decomposition approach. The method iteratively refines the model dynamically and selects regions based on uncertainty awareness, focusing computational effort exclusively on the currently most vulnerable state subsets for fine-grained modeling and optimization—thereby jointly improving accuracy and efficiency. Its core innovation lies in recasting policy synthesis as an incremental refinement process targeted at critical uncertain regions, circumventing prohibitively expensive global computations. Experiments on MDP benchmarks with over one million states demonstrate that our approach achieves up to a 2× speedup over the state-of-the-art tool PRISM, significantly enhancing the feasibility and practicality of policy synthesis for large-scale systems.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Software-intensive systems, such as software product lines and robotics, utilise Markov decision processes (MDPs) to capture uncertainty and analyse sequential decision-making problems. Despite the usefulness of conventional policy synthesis methods, they fail to scale to large state spaces. Our approach addresses this issue and accelerates policy synthesis in large MDPs by dynamically refining the MDP and iteratively selecting the most fragile MDP regions for refinement. This iterative procedure offers a balance between accuracy and efficiency, as refinement occurs only when necessary. Through a comprehensive empirical evaluation comprising diverse case studies and MDPs up to 1M states, we demonstrate significant performance improvements yielded by our approach compared to the leading probabilistic model checker PRISM (up to 2x), thus offering a very competitive solution for real-world policy synthesis tasks in larger MDPs.

Problem

• Scaling policy synthesis for large MDP state spaces
• Improving efficiency via dynamic MDP refinement
• Balancing accuracy and speed in iterative refinement

Innovation

• Hierarchical block decomposition for MDPs
• Dynamic refinement of fragile regions
• Iterative balance between accuracy and efficiency