Understanding and Stabilizing Deep Q-Learning via Controlled Bootstrapping and Regulated Value Dynamics

2026-08-17

AI Summary

This study addresses the training instability of Deep Q-Learning and the limited understanding of error interaction mechanisms by systematically analyzing multi-source instability roots. We propose a novel paradigm integrating controlled bootstrapping, ensemble quantile estimation, and spike-based parameter tuning. This approach effectively overcomes reward-induced self-reinforcing traps and parameter spiking issues, establishing a unified theoretical framework for stability. Evaluated on Atari-100K and Procgen benchmarks, the proposed method significantly enhances training stability while achieving competitive performance. These findings provide crucial theoretical and practical foundations for optimizing the robustness of reinforcement learning algorithms, offering actionable insights into mitigating complex error propagation in value-based methods.

Abstract

Deep Q-learning (DQL) has achieved remarkable empirical success in reinforcement learning, yet its training process remains notoriously unstable. Existing studies often attribute instability to isolated factors such as overestimation bias or representation learning issues, lacking a unified understanding of how different sources of instability interact during recursive value estimation. In this work, we provide a systematic analysis of instability in deep Q-learning from three complementary perspectives: operator-level bias in Bellman bootstrapping, estimator-level sensitivity of greedy action selection to regression noise, and parameter-dynamics imbalance under aggressive data reuse. We identify a reward-triggered self-reinforcing trap and characteristic parameter spike dynamics, then derive stabilization principles for controlled bootstrapping, ensemble quantile estimation, and spike-based parameter regulation. Experiments on Atari-100K and Procgen demonstrate competitive performance and improved training stability.

Problem

• Deep Q-Learning
• Training Instability
• Value Estimation
• Bootstrapping Bias
• Parameter Dynamics

Innovation

• Controlled Bootstrapping
• Ensemble Quantile Estimation
• Spike-based Parameter Regulation
• Value Dynamics
• Deep Q-Learning Stability