Uncertainty-aware Distributional Offline Reinforcement Learning

2024-03-26

arXiv.org

AI Summary

In offline reinforcement learning, policy safety is compromised by both epistemic uncertainty (model estimation error) and aleatoric uncertainty (stochasticity in state transitions and reward noise), yet existing approaches primarily address only the former. This paper introduces the first unified framework for modeling both uncertainty types in offline RL: leveraging distributional RL and uncertainty quantification, we propose a model-free algorithm that directly learns the full distribution of discounted cumulative returns; further, we integrate a conservative policy optimization mechanism that enforces risk-averse behavior at the distributional level. Evaluated on both risk-sensitive and risk-neutral benchmarks, our method consistently outperforms current state-of-the-art approaches, demonstrating that joint distributional modeling is essential for safe and robust policy learning in offline settings.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Multiagent Systems: Multiagent Systems under Uncertainty

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Offline reinforcement learning (RL) presents distinct challenges as it relies solely on observational data. A central concern in this context is ensuring the safety of the learned policy by quantifying uncertainties associated with various actions and environmental stochasticity. Traditional approaches primarily emphasize mitigating epistemic uncertainty by learning risk-averse policies, often overlooking environmental stochasticity. In this study, we propose an uncertainty-aware distributional offline RL method to simultaneously address both epistemic uncertainty and environmental stochasticity. We propose a model-free offline RL algorithm capable of learning risk-averse policies and characterizing the entire distribution of discounted cumulative rewards, as opposed to merely maximizing the expected value of accumulated discounted returns. Our method is rigorously evaluated through comprehensive experiments in both risk-sensitive and risk-neutral benchmarks, demonstrating its superior performance.

Problem

• Addresses epistemic uncertainty and environmental stochasticity in offline RL
• Learns risk-averse policies using distributional offline RL
• Evaluates performance in risk-sensitive and risk-neutral benchmarks

Innovation

• Model-free offline RL algorithm
• Addresses epistemic and environmental uncertainties
• Learns risk-averse reward distributions