Synthesizing POMDP Policies: Sampling Meets Model-checking via Learning

2026-05-14

AI Summary

This work addresses the challenge of balancing formal correctness and scalability in policy synthesis for partially observable Markov decision processes (POMDPs). The authors propose a novel framework that integrates sampling, automata learning, and model checking, introducing Angluin’s L* algorithm to POMDP controller synthesis for the first time. In this approach, sampling is employed to answer membership queries, while model checking handles equivalence queries, enabling the collaborative construction of finite-state controllers with formal guarantees. The method enjoys relative completeness in theory and demonstrates strong empirical performance, effectively solving threshold safety problems that are intractable for existing formal tools. This paradigm thus offers a scalable yet formally assured approach to POMDP policy synthesis.

Abstract

Partially Observable Markov Decision Processes (POMDPs) are the standard framework for decision-making under uncertainty. While sampling-based methods scale well, they lack formal correctness guarantees, making them unsuitable for safety-critical applications. Conversely, formal synthesis techniques provide correctness-by-construction but often struggle with scalability, as general POMDP synthesis is undecidable. To bridge this gap, we propose a synthesis framework that integrates sampling, automata learning, and model-checking. Inspired by Angluin's $L^*$ algorithm, our approach utilizes sampling as a membership oracle and model-checking as an equivalence oracle. This enables the synthesis of finite-state controllers with formal guarantees, provided the sampling-induced policy is regular. We establish a relative completeness result for this framework. Experimental results from our prototypical implementation demonstrate that this method successfully solves threshold-safety problems that remain challenging for existing formal synthesis tools. We believe our algorithm serves as a valuable component in a portfolio approach to tackling the inherent difficulty of POMDP synthesis problems.

Problem

• POMDP
• policy synthesis
• formal verification
• sampling
• model-checking

Innovation

• POMDP synthesis
• automata learning
• model-checking
• sampling-based planning
• formal guarantees