Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning

2024-11-07

arXiv.org

AI Summary

Offline reinforcement learning suffers from poor policy generalization and overestimation of out-of-distribution actions due to limited, static datasets. Method: This paper proposes a latent-space action-constrained model-based approach that jointly models the generative latent-variable distribution over observations and actions, explicitly constraining policy optimization to the support set of the latent action distribution. Contribution/Results: We introduce a novel latent action support constraint mechanism—replacing conventional ensemble-based uncertainty estimation and explicit conservatism penalties—enabling Bellman updates without explicit conservative terms and substantially reducing gradient steps required for policy optimization. Our method achieves state-of-the-art performance on both D4RL and V-D4RL benchmarks, with particularly significant gains in vision-based tasks.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Online Learning & Bandits
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In offline reinforcement learning, a policy is learned using a static dataset in the absence of costly feedback from the environment. In contrast to the online setting, only using static datasets poses additional challenges, such as policies generating out-of-distribution samples. Model-based offline reinforcement learning methods try to overcome these by learning a model of the underlying dynamics of the environment and using it to guide policy search. It is beneficial but, with limited datasets, errors in the model and the issue of value overestimation among out-of-distribution states can worsen performance. Current model-based methods apply some notion of conservatism to the Bellman update, often implemented using uncertainty estimation derived from model ensembles. In this paper, we propose Constrained Latent Action Policies (C-LAP) which learns a generative model of the joint distribution of observations and actions. We cast policy learning as a constrained objective to always stay within the support of the latent action distribution, and use the generative capabilities of the model to impose an implicit constraint on the generated actions. Thereby eliminating the need to use additional uncertainty penalties on the Bellman update and significantly decreasing the number of gradient steps required to learn a policy. We empirically evaluate C-LAP on the D4RL and V-D4RL benchmark, and show that C-LAP is competitive to state-of-the-art methods, especially outperforming on datasets with visual observations.

Problem

• Model-based Offline Reinforcement Learning
• Data Limitations
• Generalization

Innovation

• C-LAP
• Offline Reinforcement Learning
• Observation and Action Generation Models