Vectorized Online POMDP Planning

2025-10-31

AI Summary

Existing parallel online planners for partially observable Markov decision processes (POMDPs) suffer from low parallel efficiency and severe synchronization bottlenecks, hindering real-time decision-making for autonomous robots in complex partially observable environments. Method: We propose VOPP (Vectorized Online POMDP Planner), the first fully vectorized online POMDP solver. VOPP analytically decomposes the POMDP optimization objective, numerically evaluating only the expectation term; introduces a tensorized state-action-observation representation and an omnidirectional vectorized computation architecture to eliminate sequential dependencies and synchronization overhead inherent in traditional Monte Carlo tree search (MCTS); and tightly integrates online Bayesian belief updating with vectorized MCTS. Results: Experiments demonstrate that VOPP achieves over 20× speedup over the current state-of-the-art parallel online POMDP solvers while preserving policy quality, significantly advancing real-time autonomous decision-making under partial observability.

Technology Category

• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Search and Optimization: Sampling/Simulation-based Search
• Multiagent Systems: Multiagent Planning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Planning under partial observability is an essential capability of autonomous robots. The Partially Observable Markov Decision Process (POMDP) provides a powerful framework for planning under partial observability problems, capturing the stochastic effects of actions and the limited information available through noisy observations. POMDP solving could benefit tremendously from massive parallelization of today's hardware, but parallelizing POMDP solvers has been challenging. They rely on interleaving numerical optimization over actions with the estimation of their values, which creates dependencies and synchronization bottlenecks between parallel processes that can quickly offset the benefits of parallelization. In this paper, we propose Vectorized Online POMDP Planner (VOPP), a novel parallel online solver that leverages a recent POMDP formulation that analytically solves part of the optimization component, leaving only the estimation of expectations for numerical computation. VOPP represents all data structures related to planning as a collection of tensors and implements all planning steps as fully vectorized computations over this representation. The result is a massively parallel solver with no dependencies and synchronization bottlenecks between parallel computations. Experimental results indicate that VOPP is at least 20X more efficient in computing near-optimal solutions compared to an existing state-of-the-art parallel online solver.

Problem

• Planning under partial observability for autonomous robots
• Parallelizing POMDP solvers to overcome synchronization bottlenecks
• Leveraging vectorized computations for efficient expectation estimation

Innovation

• Vectorized tensor representation for planning data structures
• Fully vectorized computations eliminate synchronization bottlenecks
• Analytical optimization reduces numerical computation to expectation estimation