Intrinsic Reward Policy Optimization for Sparse-Reward Environments

2026-01-29

AI Summary

This work addresses the inefficiency of conventional exploration methods in sparse-reward environments, where existing intrinsic reward–based approaches often suffer from unstable credit assignment or insufficient sample efficiency. To overcome these limitations, we propose a novel policy optimization framework that integrates multiple intrinsic reward mechanisms to directly optimize the extrinsic reward objective, without requiring pre-trained sub-policies. By constructing a more informative surrogate policy gradient, our method enables more stable and efficient credit assignment. Empirical evaluations on both discrete and continuous sparse-reward tasks demonstrate that the proposed approach substantially outperforms state-of-the-art baselines, achieving significant improvements in both final performance and sample efficiency.

Technology Category

• Search and Optimization: Learning to Search
• Intelligent Robots: Learning & Optimization for ROB
• Machine Learning: Optimization

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Exploration is essential in reinforcement learning as an agent relies on trial and error to learn an optimal policy. However, when rewards are sparse, naive exploration strategies, like noise injection, are often insufficient. Intrinsic rewards can also provide principled guidance for exploration by, for example, combining them with extrinsic rewards to optimize a policy or using them to train subpolicies for hierarchical learning. However, the former approach suffers from unstable credit assignment, while the latter exhibits sample inefficiency and sub-optimality. We propose a policy optimization framework that leverages multiple intrinsic rewards to directly optimize a policy for an extrinsic reward without pretraining subpolicies. Our algorithm -- intrinsic reward policy optimization (IRPO) -- achieves this by using a surrogate policy gradient that provides a more informative learning signal than the true gradient in sparse-reward environments. We demonstrate that IRPO improves performance and sample efficiency relative to baselines in discrete and continuous environments, and formally analyze the optimization problem solved by IRPO. Our code is available at https://github.com/Mgineer117/IRPO.

Problem

• sparse-reward
• reinforcement learning
• exploration
• intrinsic reward
• policy optimization

Innovation

• intrinsic reward
• policy optimization
• sparse-reward reinforcement learning
• surrogate policy gradient
• sample efficiency