Scalable Option Learning in High-Throughput Environments

2025-08-29

AI Summary

To address the scalability limitations and poor support for long-horizon decision-making in hierarchical reinforcement learning (HRL) under large-scale, high-throughput settings, this paper proposes Scalable Option Learning (SOL). SOL integrates the option framework with high-throughput distributed training, leveraging massive parallel sampling and decoupled hierarchical policy optimization to significantly improve training efficiency. On NetHack, SOL achieves 20 billion frame-level training—the largest-scale HRL evaluation to date—with a throughput 25× higher than conventional HRL methods, and demonstrates, for the first time, the effectiveness of long-horizon policies in this complex environment. SOL further validates its generalizability and robustness on MiniHack and MuJoCo. The implementation is open-sourced, establishing a reproducible, scalable new paradigm for large-scale HRL.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Scalability of ML Systems
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Hierarchical reinforcement learning (RL) has the potential to enable effective decision-making over long timescales. Existing approaches, while promising, have yet to realize the benefits of large-scale training. In this work, we identify and solve several key challenges in scaling hierarchical RL to high-throughput environments. We propose Scalable Option Learning (SOL), a highly scalable hierarchical RL algorithm which achieves a 25x higher throughput compared to existing hierarchical methods. We train our hierarchical agents using 20 billion frames of experience on the complex game of NetHack, significantly surpassing flat agents and demonstrating positive scaling trends. We also validate our algorithm on MiniHack and Mujoco environments, showcasing its general applicability. Our code is open sourced at github.com/facebookresearch/sol.

Problem

• Scaling hierarchical reinforcement learning to high-throughput environments
• Achieving effective long-timescale decision-making in complex tasks
• Overcoming limitations of existing hierarchical RL methods

Innovation

• Scalable hierarchical reinforcement learning algorithm
• 25x higher throughput than existing methods
• Trained with 20 billion experience frames