Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback

2024-10-30

arXiv.org

AI Summary

To address scalability bottlenecks in sparse-reward reinforcement learning—stemming from excessive environment interactions or reliance on large offline datasets—this paper proposes ONI: an online, distributed, LLM-driven intrinsic reward synthesis framework. ONI asynchronously annotates agent-collected experiences using large language models and distills feedback into a lightweight intrinsic reward model via experience replay. Its key contribution is the first end-to-end online intrinsic reward learning pipeline that requires no offline dataset, enabling scalable deployment across billion-scale environment interactions. We systematically benchmark three modeling paradigms—hashing, classification, and ranking—yielding principled insights into intrinsic reward design. Evaluated on the sparse-reward NetHack benchmark, ONI achieves state-of-the-art performance. The full implementation is open-sourced.

Technology Category

• Machine Learning: Online Learning & Bandits
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Automatically synthesizing dense rewards from natural language descriptions is a promising paradigm in reinforcement learning (RL), with applications to sparse reward problems, open-ended exploration, and hierarchical skill design. Recent works have made promising steps by exploiting the prior knowledge of large language models (LLMs). However, these approaches suffer from important limitations: they are either not scalable to problems requiring billions of environment samples, due to requiring LLM annotations for each observation, or they require a diverse offline dataset, which may not exist or be impossible to collect. In this work, we address these limitations through a combination of algorithmic and systems-level contributions. We propose oni, a distributed architecture that simultaneously learns an RL policy and an intrinsic reward function using LLM feedback. Our approach annotates the agent's collected experience via an asynchronous LLM server, which is then distilled into an intrinsic reward model. We explore a range of algorithmic choices for reward modeling with varying complexity, including hashing, classification, and ranking models. By studying their relative tradeoffs, we shed light on questions regarding intrinsic reward design for sparse reward problems. Our approach achieves state-of-the-art performance across a range of challenging, sparse reward tasks from the NetHack Learning Environment in a simple unified process, solely using the agent's gathered experience, without requiring external datasets. We make our code available at url{https://github.com/facebookresearch/oni}.

Problem

• Automate dense reward synthesis from language descriptions
• Scale RL training without per-observation LLM queries
• Eliminate dependency on diverse offline datasets

Innovation

• Distributed architecture for RL and intrinsic rewards
• Asynchronous LLM server for experience annotation
• Diverse reward modeling with hashing, classification, ranking