🤖 AI Summary
Unsupervised skill learning often yields ineffective or hazardous robot behaviors due to the absence of implicit safety and utility constraints. To address this, we propose a two-stage instruction-driven framework: first, a discriminative network is trained on a small set of action-free instructional videos (<8 clips) to model desirable versus undesirable state transitions; second, this network is embedded as a differentiable distance metric into a DIAYN-style diversity-maximizing skill discovery objective, jointly optimizing for both safety and behavioral diversity. Our key contributions are: (i) the first preference injection mechanism grounded in action-free instructional videos, and (ii) the first direct integration of an instruction-based discriminative network into the skill learning objective function. Experiments across multiple continuous-control benchmarks demonstrate stable acquisition of complex locomotion skills—including walking and running—while significantly reducing high-risk failures such as falling and falling into pits.
📝 Abstract
Unsupervised skill discovery is a learning paradigm that aims to acquire diverse behaviors without explicit rewards. However, it faces challenges in learning complex behaviors and often leads to learning unsafe or undesirable behaviors. For instance, in various continuous control tasks, current unsupervised skill discovery methods succeed in learning basic locomotions like standing but struggle with learning more complex movements such as walking and running. Moreover, they may acquire unsafe behaviors like tripping and rolling or navigate to undesirable locations such as pitfalls or hazardous areas. In response, we present DoDont (Do's and Don'ts), an instruction-based skill discovery algorithm composed of two stages. First, in an instruction learning stage, DoDont leverages action-free instruction videos to train an instruction network to distinguish desirable transitions from undesirable ones. Then, in the skill learning stage, the instruction network adjusts the reward function of the skill discovery algorithm to weight the desired behaviors. Specifically, we integrate the instruction network into a distance-maximizing skill discovery algorithm, where the instruction network serves as the distance function. Empirically, with less than 8 instruction videos, DoDont effectively learns desirable behaviors and avoids undesirable ones across complex continuous control tasks. Code and videos are available at https://mynsng.github.io/dodont/