🤖 AI Summary
In multi-timescale reinforcement learning, naively aggregating rewards under different discount factors often leads to surrogate objective misuse and myopic degradation. This work proposes a Target Decoupling architecture that preserves multi-scale value predictions in the critic to enhance representation learning while restricting the actor’s policy updates exclusively to long-horizon advantages, thereby preventing interference from short-term signals. By decoupling the policy and value update pathways, the approach effectively circumvents these pitfalls. Implemented within a PPO framework and augmented with multiple discount factors, auxiliary representation learning, and gradient isolation mechanisms, the method consistently surpasses the “solved” threshold on LunarLander-v2, significantly improving performance, eliminating policy collapse, and escaping local optima inherent to single-timescale approaches.
📝 Abstract
Temporal credit assignment in reinforcement learning has long been a central challenge. Inspired by the multi-timescale encoding of the dopamine system in neurobiology, recent research has sought to introduce multiple discount factors into Actor-Critic architectures, such as Proximal Policy Optimization (PPO), to balance short-term responses with long-term planning. However, this paper reveals that blindly fusing multi-timescale signals in complex delayed-reward tasks can lead to severe algorithmic pathologies. We systematically demonstrate that exposing a temporal attention routing mechanism to policy gradients results in surrogate objective hacking, while adopting gradient-free uncertainty weighting triggers irreversible myopic degeneration, a phenomenon we term the Paradox of Temporal Uncertainty. To address these issues, we propose a Target Decoupling architecture: on the Critic side, we retain multi-timescale predictions to enforce auxiliary representation learning, while on the Actor side, we strictly isolate short-term signals and update the policy based solely on long-term advantages. Rigorous empirical evaluations across multiple independent random seeds in the LunarLander-v2 environment demonstrate that our proposed architecture achieves statistically significant performance improvements. Without relying on hyperparameter hacking, it consistently surpasses the ''Environment Solved'' threshold with minimal variance, completely eliminates policy collapse, and escapes the hovering local optima that trap single-timescale baselines.