Autotelic Reinforcement Learning: Exploring Intrinsic Motivations for Skill Acquisition in Open-Ended Environments

2025-02-06

AI Summary

This paper addresses autonomous skill acquisition in open-ended environments without external rewards. Methodologically, it proposes the autotelic reinforcement learning framework, driven by intrinsic motivation—comprising epistemic and competence-based components—to enable spontaneous goal generation, skill representation learning, and mastery assessment. It formally defines the autotelic RL paradigm; establishes a taxonomy of IMGEPs-based goal-generation mechanisms; models skill acquisition as a self-guided, reward-free process within an MDP; and introduces a unified evaluation metric integrating exploration, generalization, and robustness. Contributions include: (i) the first benchmark for quantitatively evaluating agent autonomy and open-ended skill growth; and (ii) empirical validation demonstrating that the framework enables sustained development of multi-goal policies and cross-task generalization in complex, dynamic environments.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Multiagent Systems: Multiagent Learning
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This paper presents a comprehensive overview of autotelic Reinforcement Learning (RL), emphasizing the role of intrinsic motivations in the open-ended formation of skill repertoires. We delineate the distinctions between knowledge-based and competence-based intrinsic motivations, illustrating how these concepts inform the development of autonomous agents capable of generating and pursuing self-defined goals. The typology of Intrinsically Motivated Goal Exploration Processes (IMGEPs) is explored, with a focus on the implications for multi-goal RL and developmental robotics. The autotelic learning problem is framed within a reward-free Markov Decision Process (MDP), WHERE agents must autonomously represent, generate, and master their own goals. We address the unique challenges in evaluating such agents, proposing various metrics for measuring exploration, generalization, and robustness in complex environments. This work aims to advance the understanding of autotelic RL agents and their potential for enhancing skill acquisition in a diverse and dynamic setting.

Problem

• Intrinsic motivations in skill acquisition
• Autonomous goal generation in RL
• Evaluating exploration in complex environments

Innovation

• Intrinsic motivations for skill acquisition
• Autonomous goal representation in open-ended environments
• Reward-free Markov Decision Process framework