Constrained and Robust Policy Synthesis with Satisfiability-Modulo-Probabilistic-Model-Checking

2025-11-11

AI Summary

This work addresses the synthesis of reward-optimal policies for finite Markov decision processes (MDPs) that simultaneously satisfy structural constraints—such as policy conciseness or implementation cost—and exhibit robustness against model perturbations. We propose the first robust policy synthesis framework supporting arbitrary first-order logic–expressible structural constraints. Our approach tightly integrates Satisfiability Modulo Theories (SMT) solving with probabilistic model checking to jointly optimize constraint satisfaction and robust performance. Crucially, it retains computational tractability while significantly enhancing the reliability of synthesized policies in practical deployment. Experimental evaluation on hundreds of benchmark instances demonstrates competitive performance across diverse constraint classes, substantially advancing both the solvability and practical applicability of structured robust policy synthesis.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The ability to compute reward-optimal policies for given and known finite Markov decision processes (MDPs) underpins a variety of applications across planning, controller synthesis, and verification. However, we often want policies (1) to be robust, i.e., they perform well on perturbations of the MDP and (2) to satisfy additional structural constraints regarding, e.g., their representation or implementation cost. Computing such robust and constrained policies is indeed computationally more challenging. This paper contributes the first approach to effectively compute robust policies subject to arbitrary structural constraints using a flexible and efficient framework. We achieve flexibility by allowing to express our constraints in a first-order theory over a set of MDPs, while the root for our efficiency lies in the tight integration of satisfiability solvers to handle the combinatorial nature of the problem and probabilistic model checking algorithms to handle the analysis of MDPs. Experiments on a few hundred benchmarks demonstrate the feasibility for constrained and robust policy synthesis and the competitiveness with state-of-the-art methods for various fragments of the problem.

Problem

• Computing robust policies for perturbed Markov decision processes
• Satisfying structural constraints on policy representation and cost
• Integrating satisfiability solvers with probabilistic model checking algorithms

Innovation

• Integrates satisfiability solvers with probabilistic model checking
• Expresses constraints in first-order theory over MDPs
• Computes robust policies under arbitrary structural constraints