Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning

2024-06-06

arXiv.org

AI Summary

To address slow convergence, high bias in value estimation, and inefficient uncertainty modeling caused by Monte Carlo sampling in offline reinforcement learning, this paper proposes a deterministic Bellman target estimation method. The core innovation lies in the first integration of progressive moment matching into model-driven offline RL, enabling deterministic approximation of state uncertainty propagation within the Q-network. By combining moment-matching variational inference with Gaussian latent-layer activation distribution modeling, the approach mitigates value overestimation under distributional shift. Theoretically, we prove that our method yields a tighter suboptimality upper bound compared to standard Monte Carlo–based approaches. Empirically, it achieves significantly faster convergence across multiple benchmark offline RL tasks—outperforming existing MC-based methods—while maintaining rigorous theoretical foundations and practical efficiency.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization
• Intelligent Robots: State Estimation

Application Category

• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Current approaches to model-based offline reinforcement learning often incorporate uncertainty-based reward penalization to address the distributional shift problem. These approaches, commonly known as pessimistic value iteration, use Monte Carlo sampling to estimate the Bellman target to perform temporal difference-based policy evaluation. We find out that the randomness caused by this sampling step significantly delays convergence. We present a theoretical result demonstrating the strong dependency of suboptimality on the number of Monte Carlo samples taken per Bellman target calculation. Our main contribution is a deterministic approximation to the Bellman target that uses progressive moment matching, a method developed originally for deterministic variational inference. The resulting algorithm, which we call Moment Matching Offline Model-Based Policy Optimization (MOMBO), propagates the uncertainty of the next state through a nonlinear Q-network in a deterministic fashion by approximating the distributions of hidden layer activations by a normal distribution. We show that it is possible to provide tighter guarantees for the suboptimality of MOMBO than the existing Monte Carlo sampling approaches. We also observe MOMBO to converge faster than these approaches in a large set of benchmark tasks.

Problem

• Offline Reinforcement Learning
• Pessimistic Value Iteration
• Monte Carlo Sampling

Innovation

• MOMBO
• Stepwise Moment Matching
• Offline Reinforcement Learning