A Provably Efficient Option-Based Algorithm for both High-Level and Low-Level Learning

2024-06-21

RLJ

AI Summary

This work addresses the lack of theoretical guarantees for joint learning of high- and low-level policies in hierarchical reinforcement learning (HRL). We propose the first provably efficient meta-algorithm based on alternating optimization. Within a finite horizon, it separately minimizes regret for the high-level semi-Markov decision process (SMDP) policy and the low-level option policies—without requiring pre-trained options. Our framework is the first to establish a tight, unified theoretical analysis for synchronous hierarchical learning, explicitly characterizing sufficient conditions under which hierarchical decomposition yields provable benefits. Moreover, we derive a rigorous advantage bound demonstrating strict improvement over non-hierarchical lower bounds. The method integrates the options framework, SMDP modeling, and online regret minimization techniques, yielding the first jointly convergent HRL algorithm with provable acceleration.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

Hierarchical Reinforcement Learning (HRL) approaches have shown successful results in solving a large variety of complex, structured, long-horizon problems. Nevertheless, a full theoretical understanding of this empirical evidence is currently missing. In the context of the emph{option} framework, prior research has devised efficient algorithms for scenarios where options are fixed, and the high-level policy selecting among options only has to be learned. However, the fully realistic scenario in which both the high-level and the low-level policies are learned is surprisingly disregarded from a theoretical perspective. This work makes a step towards the understanding of this latter scenario. Focusing on the finite-horizon problem, we present a meta-algorithm alternating between regret minimization algorithms instanced at different (high and low) temporal abstractions. At the higher level, we treat the problem as a Semi-Markov Decision Process (SMDP), with fixed low-level policies, while at a lower level, inner option policies are learned with a fixed high-level policy. The bounds derived are compared with the lower bound for non-hierarchical finite-horizon problems, allowing to characterize when a hierarchical approach is provably preferable, even without pre-trained options.

Problem

• Hierarchical Reinforcement Learning theory gap
• Simultaneous high and low-level policy learning
• Provable efficiency in finite-horizon problems

Innovation

• Hierarchical Reinforcement Learning approach
• Meta-algorithm with regret minimization
• Semi-Markov Decision Process framework