CODE-SHARP: Continuous Open-ended Discovery and Evolution of Skills as Hierarchical Reward Programs

2026-02-10

AI Summary

This work addresses the challenge of enabling agents to continuously discover and learn new skills in the absence of a predefined task set. It proposes the first foundation model–based framework that constructs a directed graph of executable reward functions to automatically generate and evolve hierarchical reward programs, thereby progressively expanding the agent’s skill repertoire. By integrating goal-conditioned reinforcement learning with a high-level planner, the approach facilitates autonomous skill discovery, composition, and optimization within the Craftax environment. Experimental results demonstrate that the learned agent outperforms both pretrained models and task-specific expert policies by over 134% on average in long-horizon tasks, substantially enhancing the efficiency of solving complex, temporally extended challenges.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Search and Optimization: Learning to Search
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Developing agents capable of open-endedly discovering and learning novel skills is a grand challenge in Artificial Intelligence. While reinforcement learning offers a powerful framework for training agents to master complex skills, it typically relies on hand-designed reward functions. This is infeasible for open-ended skill discovery, where the set of meaningful skills is not known a priori. While recent methods have shown promising results towards automating reward function design, they remain limited to refining rewards for pre-defined tasks. To address this limitation, we introduce Continuous Open-ended Discovery and Evolution of Skills as Hierarchical Reward Programs (CODE-SHARP), a novel framework leveraging Foundation Models (FM) to open-endedly expand and refine a hierarchical skill archive, structured as a directed graph of executable reward functions in code. We show that a goal-conditioned agent trained exclusively on the rewards generated by the discovered SHARP skills learns to solve increasingly long-horizon goals in the Craftax environment. When composed by a high-level FM-based planner, the discovered skills enable a single goal-conditioned agent to solve complex, long-horizon tasks, outperforming both pretrained agents and task-specific expert policies by over $134$% on average. We will open-source our code and provide additional videos at https://sites.google.com/view/code-sharp/homepage.

Problem

• open-ended skill discovery
• reward function design
• hierarchical skills
• reinforcement learning
• Foundation Models

Innovation

• open-ended skill discovery
• hierarchical reward programs
• foundation models
• reinforcement learning
• long-horizon tasks