Unsupervised Hierarchical Skill Discovery

2026-01-30

AI Summary

This work addresses the challenge of automatically discovering reusable skills and their hierarchical structure from raw trajectories without any labels, rewards, or action supervision. To this end, we propose an unsupervised grammar induction–based approach that, for the first time, enables hierarchical skill segmentation and modeling in the complete absence of supervisory signals. Our method effectively identifies low-level behavioral primitives and their compositional higher-level skills in high-dimensional pixel-based environments such as Craftax and vanilla Minecraft, yielding substantially improved semantic clarity and structural coherence. Experimental results demonstrate that the discovered skills outperform existing baselines in segmentation accuracy, reusability, and hierarchical quality, and further facilitate faster and more stable training in downstream reinforcement learning tasks.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

We consider the problem of unsupervised skill segmentation and hierarchical structure discovery in reinforcement learning. While recent approaches have sought to segment trajectories into reusable skills or options, most rely on action labels, rewards, or handcrafted annotations, limiting their applicability. We propose a method that segments unlabelled trajectories into skills and induces a hierarchical structure over them using a grammar-based approach. The resulting hierarchy captures both low-level behaviours and their composition into higher-level skills. We evaluate our approach in high-dimensional, pixel-based environments, including Craftax and the full, unmodified version of Minecraft. Using metrics for skill segmentation, reuse, and hierarchy quality, we find that our method consistently produces more structured and semantically meaningful hierarchies than existing baselines. Furthermore, as a proof of concept for utility, we demonstrate that these discovered hierarchies accelerate and stabilise learning on downstream reinforcement learning tasks.

Problem

• unsupervised skill discovery
• hierarchical structure
• skill segmentation
• reinforcement learning
• trajectory analysis

Innovation

• unsupervised skill discovery
• hierarchical reinforcement learning
• grammar-based segmentation
• skill composition
• trajectory segmentation