Language Guided Skill Discovery

2024-06-07

arXiv.org

AI Summary

To address weak generalization in skill discovery caused by insufficient semantic diversity, this paper proposes an LLM-guided prompt-constrained skill learning framework. Methodologically, it is the first to explicitly maximize semantic dissimilarity among skills by directly leveraging large language models’ semantic priors; skills are learned within user-defined natural-language prompts’ semantic subspaces via joint optimization of reward-free reinforcement learning and a skill discriminator, enabling semantic-aware state-space exploration. Contributions include: (1) establishing an interpretable mapping from natural language to the skill semantic space; and (2) supporting plug-and-play, prompt-driven downstream task adaptation. Evaluated on legged robot navigation and robotic arm manipulation tasks, the method significantly outperforms five baselines. Crucially, altering only the input prompt activates distinct semantic behavior modes—demonstrating strong generalization and precise controllability.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search
• Intelligent Robots: Behavior Learning & Control

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Skill discovery methods enable agents to learn diverse emergent behaviors without explicit rewards. To make learned skills useful for unknown downstream tasks, obtaining a semantically diverse repertoire of skills is essential. While some approaches introduce a discriminator to distinguish skills and others aim to increase state coverage, no existing work directly addresses the"semantic diversity"of skills. We hypothesize that leveraging the semantic knowledge of large language models (LLMs) can lead us to improve semantic diversity of resulting behaviors. In this sense, we introduce Language Guided Skill Discovery (LGSD), a skill discovery framework that aims to directly maximize the semantic diversity between skills. LGSD takes user prompts as input and outputs a set of semantically distinctive skills. The prompts serve as a means to constrain the search space into a semantically desired subspace, and the generated LLM outputs guide the agent to visit semantically diverse states within the subspace. We demonstrate that LGSD enables legged robots to visit different user-intended areas on a plane by simply changing the prompt. Furthermore, we show that language guidance aids in discovering more diverse skills compared to five existing skill discovery methods in robot-arm manipulation environments. Lastly, LGSD provides a simple way of utilizing learned skills via natural language.

Problem

• Improve semantic diversity in skill discovery
• Leverage LLMs for semantically diverse behaviors
• Enable robots to perform tasks via natural language prompts

Innovation

• Leverages LLMs for semantic diversity
• Uses prompts to guide skill discovery
• Enables robots via natural language commands