Offline Policy Optimization with Posterior Sampling

2026-05-08

AI Summary

This work addresses the fundamental trade-off in offline reinforcement learning between generalization capability and robustness to out-of-distribution model errors by proposing Posterior Sampling Policy Optimization (PSPO). PSPO treats dynamics modeling as a Bayesian inference problem, integrating posterior sampling with constrained policy optimization to explicitly quantify model fidelity. This approach enables the safe utilization of out-of-distribution yet dynamics-consistent data while preserving robustness. Theoretical analysis establishes the convergence of Q-value estimates and monotonic policy improvement, overcoming the limitations of conventional overly pessimistic regularization schemes. Empirical evaluations demonstrate that PSPO significantly outperforms state-of-the-art methods on standard offline RL benchmarks.

Abstract

A fundamental challenge in model-based offline reinforcement learning (RL) lies in the trade-off between generalization and robustness against exploitation errors in out-of-distribution (OOD) regions. While OOD samples may capture valid underlying physical dynamics, they also introduce the risk of model exploitation. Existing methods typically address this risk through excessive pessimistic regularization, which ensures robustness but often sacrifices generalization. To overcome this limitation, we propose Posterior Sampling-based Policy Optimization (PSPO), which formulates dynamics modeling as a Bayesian inference process to derive a posterior that explicitly quantifies model fidelity. Through the integration of posterior sampling and constrained policy optimization, our method leverages dynamics-consistent OOD transitions for generalization while ensuring robustness against model exploitation. Theoretically, we formulate Q-value estimation under posterior sampling as a stochastic approximation problem and establish its convergence. We decompose policy optimization into a sequence of constrained subproblems, demonstrating that solving these subproblems guarantees monotonic improvement until convergence. Experiments on standard benchmarks validate that PSPO achieves superior performance compared to state-of-the-art baselines.

Problem

• offline reinforcement learning
• out-of-distribution
• generalization
• model exploitation
• robustness

Innovation

• posterior sampling
• offline reinforcement learning
• model-based RL
• out-of-distribution generalization
• constrained policy optimization