Generative OOD-regularized Model-based Policy Optimization

2026-05-23

AI Summary

This work addresses the challenge in offline reinforcement learning where sparse state-action distributions often lead policies to generate out-of-distribution (OOD) actions, compromising safety. To mitigate this, the paper proposes GORMPO, the first algorithm to systematically integrate generative density estimation with model-based policy optimization. GORMPO constrains policy updates to high-data-density regions and incorporates both OOD regularization and conservative penalty mechanisms to effectively suppress unsafe OOD behavior. Theoretical analysis provides performance guarantees and reveals a dependency between OOD detection capability and policy performance, modulated by the stability of environment dynamics. Empirical results demonstrate that GORMPO significantly enhances baseline model performance on standard offline RL benchmarks and outperforms state-of-the-art methods by 17% on a real-world medical dataset.

Abstract

We study sequential decision-making with offline reinforcement learning (RL). Traditional offline RL policies may result in out-of-distribution (OOD) actions when training relies only on sparse offline representations. To ensure safe offline policies in a sparse state-action space, we explore how density estimation models can be integrated into model-based RL methods to avoid the OOD regions. Generative models are capable of explicitly modeling the density in sparse state-action spaces. Building on this, we introduce Generative OOD-regularized Model-based Policy Optimization (GORMPO), a density-regularized offline RL algorithm that uses generative density modeling to restrict policy updates to high-density areas of the dataset. Furthermore, we examine whether better OOD detection corresponds to better model-based offline policies. We compare (1) the OOD detection capabilities of various density estimators and (2) their performance within the GORMPO framework on a real-world medical dataset and sparse offline RL datasets. We theoretically guarantee GORMPO's performance under mild assumptions. Empirically, GORMPO outperforms state-of-the-art baselines by 17% on a real-world medical dataset and enhances the base model on the offline RL datasets. Our empirical findings show that better OOD detection generally results in improved policies in environments with stable dynamics, while conservative penalties with poor density estimation are favored when dynamics are uncertain.

Problem

• offline reinforcement learning
• out-of-distribution
• sparse state-action space
• policy safety
• distributional shift

Innovation

• generative modeling
• out-of-distribution detection
• offline reinforcement learning
• density estimation
• model-based policy optimization