Problem
Research questions and friction points this paper is trying to address.
Batch Normalization
value-based reinforcement learning
discrete-action value learning
On BatchNorm Forward Modes in Value-Based Reinforcement Learning
🤖 AI Summary
研究解决了BN在离散动作价值学习中的性能下降问题，通过选择合适的前向统计模式（如批量统计）来提高C51和PQN算法的表现。
📝 Abstract
Batch normalization (BN) substantially improves sample efficiency in continuous-control actor-critic methods such as CrossQ, yet recent studies report performance degradation in discrete-action value learning on Atari. These failures are surprising because discrete Q-networks lack the action-input distribution mismatch identified by CrossQ. We show for target-based C51 and target-free PQN that the simple choice between running and batch statistics at specific forward passes can reverse this degradation. In C51, switching the BN bootstrap forward to batch-statistic mode significantly improves performance over unnormalized and LayerNorm baselines and scales stably with update-to-data ratios up to 12. In PQN, using batch-statistics for both action selection and bootstrapping recovers performance from the failing running-statistic configuration. Across 26 Atari games at 400M frames, this configuration achieves a higher final aggregate score than PQN with LayerNorm. Our results show that carefully configured BN can substantially improve discrete-action value learning, and that its forward protocols are an essential part of the algorithm specification.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Batch Normalization
Discrete-action Value Learning
Running Statistics
Batch Statistics
C51 and PQN
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
