POPGym Arcade: Parallel Pixelated POMDPs

2025-03-03

AI Summary

This work addresses three key challenges in POMDP benchmarking: the lack of standardization, difficulties in evaluating agents in pixel-based environments, and low training efficiency. To this end, we introduce the first scalable, pixel-based POMDP benchmark supporting both fully and partially observable settings, with seven parallel tasks spanning three difficulty levels. Methodologically, we propose a Just-In-Time (JIT)-compiled, Podracer-inspired parallel architecture enabling hardware acceleration and high memory throughput; design pixel-level observation modeling with memory-augmented Q-learning; and pioneer memory saliency maps to visualize memory propagation within learned policies. Our contributions are: (1) the first open-source, extensible pixel-based POMDP benchmark; (2) empirical validation that memory mechanisms are decisive for generalization across tasks and observability conditions; and (3) substantial improvements in training speed and hardware utilization.

Technology Category

• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Search and Optimization: Learning to Search
• Multiagent Systems: Adversarial Agents

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We introduce POPGym Arcade, a benchmark consisting of 7 pixel-based environments each with three difficulties, utilizing a single observation and action space. Each environment offers both fully observable and partially observable variants, enabling counterfactual studies on partial observability. POPGym Arcade utilizes JIT compilation on hardware accelerators to achieve substantial speedups over CPU-bound environments. Moreover, this enables Podracer-style architectures to further increase hardware utilization and training speed. We evaluate memory models on our environments using a Podracer variant of Q learning, and examine the results. Finally, we generate memory saliency maps, uncovering how memories propagate through policies. Our library is available at https://github.com/bolt-research/popgym_arcade.

Problem

• Benchmark for pixel-based environments with varying difficulties.
• Enables counterfactual studies on partial observability in environments.
• Utilizes JIT compilation for speedups and examines memory models.

Innovation

• JIT compilation on hardware accelerators
• Podracer-style architectures for training
• Memory saliency maps for policy analysis