Accelerating Q-learning through Efficient Value-Sharing across Actions

2026-06-29

AI Summary

Traditional Q-learning suffers from slow convergence and value overestimation due to its independent updates of individual state-action pairs and the absence of value sharing across actions. This work proposes a parameter-free mean-expansion layer that, without altering the underlying algorithmic framework, enables cross-action value sharing within each state and reformulates the learning objective into a low-norm representation to facilitate efficient action-value estimation. For the first time, this architectural component is integrated into mainstream algorithms such as DQN and IQN, yielding substantial performance gains across 57 Atari games. The method effectively enlarges the value gap between optimal and suboptimal actions and significantly mitigates the overestimation bias inherent in conventional approaches.

Abstract

Action-values are foundational to many control algorithms such as Q-learning. Therefore learning action-values efficiently is central to reinforcement learning (RL). However, learning them can be slow, requiring many updates to move values from their initialization, typically near zero, to their true values, which may be far from zero. Moreover, action-value learning algorithms typically update each state-action pair independently, without learning shared value structure across actions within a state. In this paper, we address these inefficiencies by introducing the mean-expansion layer, which accelerates action-value learning by sharing values across actions within a state and by changing the problem from directly learning potentially large action-values to learning a lower-norm representation of them. In deep RL, this layer can be applied as a parameter-free addition to Q-network architectures without altering the underlying algorithm. Applied to deep Q-networks and implicit quantile networks, it improves aggregate performance across 57 Atari games while increasing action gaps and dramatically reducing value overestimation.

Problem

• Q-learning
• action-values
• value-sharing
• reinforcement learning
• learning efficiency

Innovation

• mean-expansion layer
• value-sharing
• Q-learning acceleration
• action-value representation
• deep reinforcement learning