TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning

2024-10-12

arXiv.org

AI Summary

To address the challenges of off-policy training difficulty, low sample efficiency, and unstable policy updates in episodic reinforcement learning (ERL), this paper proposes the first ERL framework supporting trajectory-level off-policy updates. Methodologically, it introduces three key innovations: (1) the first integration of Transformers into the critic architecture for ERL, enabling segmented modeling of long-horizon action trajectories; (2) a hybrid value estimation scheme combining *n*-step returns with segment-wise sequence value prediction, thereby relaxing strict on-policy constraints; and (3) policy parameterization via movement primitives to enhance interpretability and generalization. Evaluated on complex robotic control benchmarks, the framework achieves significant improvements over state-of-the-art methods. Ablation studies quantitatively validate the individual contributions of each component to sample efficiency, convergence stability, and final performance.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Intelligent Robots: Learning & Optimization for ROB
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This work introduces Transformer-based Off-Policy Episodic Reinforcement Learning (TOP-ERL), a novel algorithm that enables off-policy updates in the ERL framework. In ERL, policies predict entire action trajectories over multiple time steps instead of single actions at every time step. These trajectories are typically parameterized by trajectory generators such as Movement Primitives (MP), allowing for smooth and efficient exploration over long horizons while capturing high-level temporal correlations. However, ERL methods are often constrained to on-policy frameworks due to the difficulty of evaluating state-action values for entire action sequences, limiting their sample efficiency and preventing the use of more efficient off-policy architectures. TOP-ERL addresses this shortcoming by segmenting long action sequences and estimating the state-action values for each segment using a transformer-based critic architecture alongside an n-step return estimation. These contributions result in efficient and stable training that is reflected in the empirical results conducted on sophisticated robot learning environments. TOP-ERL significantly outperforms state-of-the-art RL methods. Thorough ablation studies additionally show the impact of key design choices on the model performance.

Problem

• Reinforcement Learning
• Learning Efficiency
• Complex Tasks

Innovation

• Transformer-based Reinforcement Learning
• Task Decomposition
• Enhanced Learning Stability and Speed