VOiLA: Vectorized Online Planning with Learned Diffusion Model for POMDP Agents

2026-06-17

AI Summary

This work addresses the challenge of online planning in real-world partially observable Markov decision processes (POMDPs), where accurate model specification is often infeasible. The authors propose a task-agnostic learning framework that, for the first time, employs conditional diffusion models to jointly learn the state transition and observation generation dynamics of a POMDP, integrating particle filtering for belief updating. By distilling the diffusion model into a lightweight surrogate, they achieve nearly three orders of magnitude speedup in sampling and leverage GPU parallelization to implement an efficient vectorized online POMDP planner (VOPP). Experiments demonstrate that the method matches or exceeds the performance of Recurrent SAC on three benchmark tasks using less than 10% of the training data and exhibits superior generalization in unseen environments. Notably, physical robot trials achieved a 100% success rate across ten tasks.

Abstract

Planning under uncertainty is an essential capability for autonomous robots. The Partially Observable Markov Decision Process (POMDP) provides a powerful framework for such a capability. Although POMDP-based planning has advanced significantly, its application to real-world problems is often limited by the difficulty of obtaining faithful POMDP models. We present Vectorized Online planning wIth Learned diffusion model for POMDP Agents (VOiLA), a framework that learns task-agnostic POMDP models for online planning under uncertainty. VOiLA learns transition and observation samplers using conditional diffusion models and learns observation-likelihood models for particle-based belief updates. To enable efficient online planning, the diffusion samplers are distilled into compact feedforward generators and integrated with Vectorized Online POMDP Planner (VOPP), an online POMDP planner designed to leverage GPU parallelization. Experimental results indicate the distillation strategy reduces sampling cost by up to nearly three orders of magnitude, making learned generative POMDP models practical for online planning. Evaluation of VOiLA on three benchmark problems indicate that VOiLA achieves equal or better performance than Recurrent Soft Actor Critic while using less than 10% training data, and generalizes much better to unseen environment configurations. Physical robot evaluation indicates VOiLA uses the models learned using only simulated data and generates a policy that successfully accomplish the task in 10 of 10 runs.

Problem

• POMDP
• online planning
• model learning
• uncertainty
• autonomous robots

Innovation

• diffusion model
• POMDP
• online planning
• model distillation
• GPU vectorization