Learning Generalizable Skill Policy with Data-Efficient Unsupervised RL

2026-06-30

AI Summary

This work addresses the fragility of downstream task generalization in unsupervised reinforcement learning, which often stems from the non-stationarity of skill semantics. To mitigate this issue, the paper proposes the GenDa framework, which introduces a novel skill relabeling mechanism to enhance pretraining data efficiency and alleviate non-stationarity. Additionally, it incorporates a complementary information bottleneck that guides the policy to focus on egocentric features, thereby improving robustness to distribution shifts. By integrating these components with a skill-conditioned policy and operating without any additional supervision, GenDa significantly advances the scalability, data efficiency, and generalization performance of unsupervised reinforcement learning across downstream tasks.

Abstract

Unsupervised Reinforcement Learning (URL) aims to pre-train scalable, skill-conditioned policies without extrinsic rewards, serving as a foundation for downstream control tasks. Despite recent progress, we argue that current off-policy URL methods are limited by two critical, overlooked bottlenecks: (1) non-stationary skill semantics and (2) brittle generalization. To address these challenges, we propose GenDa (Generalizable Data-efficient Agent), a unified framework for robust unsupervised reinforcement learning. First, we introduce a skill relabeling mechanism to mitigate non-stationarity and significantly improve data efficiency for pre-training. Second, we propose a Complementary Information Bottleneck (CIB), encouraging the learned skill policy to focus on ego-centric features and become robust to distribution shifts for downstream tasks. Through various experiments, we demonstrate that GenDa significantly enhances the scalability of URL with superior generalizability and data efficiency. Our code and videos are available at https://ihatebroccoli.github.io/official-GenDa.

Problem

• Unsupervised Reinforcement Learning
• Skill Policy
• Non-stationary Skill Semantics
• Generalization
• Data Efficiency

Innovation

• Unsupervised Reinforcement Learning
• Skill Relabeling
• Information Bottleneck
• Data Efficiency
• Generalization