Hierarchical Behaviour Spaces

2026-04-27

AI Summary

This work addresses the limited policy expressivity and insufficient exploration efficiency in hierarchical reinforcement learning by proposing a novel paradigm in which a controller constructs a flexible behavioral space through linear combinations of multiple option reward functions. Departing from conventional assumptions of long-horizon planning, the approach demonstrates that the core advantage of hierarchical structures lies in their capacity to enhance exploration. Empirical evaluation in the NetHack Learning Environment shows that the proposed method substantially improves both exploration efficiency and overall learning performance, thereby validating its effectiveness and superiority in complex environments.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Search and Optimization: Learning to Search
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

Recent work in hierarchical reinforcement learning has shown success in scaling to billions of timesteps when learning over a set of predefined option reward functions. We show that, instead of using a single reward function per option, the reward functions can be effectively used to induce a space of behaviours, by letting the controller specify linear combinations over reward functions, allowing a more expressive set of policies to be represented. We call this method Hierarchical Behaviour Spaces (HBS). We evaluate HBS on the NetHack Learning Environment, demonstrating strong performance. We conduct a series of experiments and determine that, perhaps going against conventional wisdom, the benefits of hierarchy in our method come from increased exploration rather than long term reasoning.

Problem

• hierarchical reinforcement learning
• behaviour spaces
• reward functions
• exploration
• policy representation

Innovation

• Hierarchical Reinforcement Learning
• Behaviour Spaces
• Reward Function Composition
• Exploration
• Option Framework