Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learning

2026-06-18

AI Summary

This work addresses the limitations of Direct Advantage Estimation (DAE) in partially observable environments and its high computational cost under high-dimensional observations. The paper presents the first extension of DAE to the Partially Observable Markov Decision Process (POMDP) setting, introducing a lightweight discrete latent-variable dynamics model to efficiently approximate state transition probabilities. This approach substantially reduces computational complexity while improving sample efficiency and scalability. Experimental results on the Arcade Learning Environment demonstrate that the proposed algorithm maintains high sample efficiency and effectively scales with the capacity of the function approximator, thereby validating its practicality and superiority in complex partially observable tasks.

Abstract

Direct Advantage Estimation (DAE) has been shown to improve the sample efficiency of deep reinforcement learning algorithms. However, its reliance on full environment observability limits its applicability in realistic settings, and its requirement to model transition probabilities incurs substantial computational overhead for high-dimensional observations. In the present work, we address both limitations. First, we extend the theoretical framework of DAE to partially observable domains with minimal modifications. Second, we reduce its computational complexity by introducing discrete latent dynamics models that efficiently approximate transition probabilities. We evaluate our approach on the Arcade Learning Environment and find that DAE scales effectively with function approximator capacity while retaining high sample efficiency.

Problem

• Direct Advantage Estimation
• sample efficiency
• partial observability
• computational overhead
• transition probability

Innovation

• Direct Advantage Estimation
• partially observable environments
• discrete latent dynamics
• sample efficiency
• scalable reinforcement learning