World Models Increase Autonomy in Reinforcement Learning

2024-08-19

arXiv.org

AI Summary

This work addresses key bottlenecks in reset-free reinforcement learning (RFRL)—namely, frequent human intervention and reliance on environmental rewards or expert demonstrations—by proposing MoReFree, a model-driven agent. Methodologically, it is the first to systematically validate the critical role of world models in reset-free settings; it introduces a decoupled framework integrating task-aware state-prioritized sampling, unsupervised intrinsic motivation for exploration, and dynamic policy updating, enabling adaptive co-optimization of exploration and policy learning. In terms of contributions and results, MoReFree achieves significantly improved data efficiency across diverse reset-free tasks, outperforming reward- or demonstration-dependent state-of-the-art baselines. It substantially reduces human reset interventions and supervision overhead, establishing a novel paradigm for autonomous, continual learning.

Technology Category

• Machine Learning: Reinforcement Learning
• Reasoning under Uncertainty: Sequential Decision Making
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Reinforcement learning (RL) is an appealing paradigm for training intelligent agents, enabling policy acquisition from the agent's own autonomously acquired experience. However, the training process of RL is far from automatic, requiring extensive human effort to reset the agent and environments. To tackle the challenging reset-free setting, we first demonstrate the superiority of model-based (MB) RL methods in such setting, showing that a straightforward adaptation of MBRL can outperform all the prior state-of-the-art methods while requiring less supervision. We then identify limitations inherent to this direct extension and propose a solution called model-based reset-free (MoReFree) agent, which further enhances the performance. MoReFree adapts two key mechanisms, exploration and policy learning, to handle reset-free tasks by prioritizing task-relevant states. It exhibits superior data-efficiency across various reset-free tasks without access to environmental reward or demonstrations while significantly outperforming privileged baselines that require supervision. Our findings suggest model-based methods hold significant promise for reducing human effort in RL. Website: https://sites.google.com/view/morefree

Problem

• Reset-free Reinforcement Learning
• Model-based RL methods
• Reducing human effort in RL

Innovation

• Model-based RL adaptation
• MoReFree agent introduction
• Task-relevant state prioritization