🤖 AI Summary
Conventional chaos-based reinforcement learning (CBRL) suffers from algorithmic lag and poor adaptability to deterministic continuous action spaces. Method: This paper proposes Chaos-enhanced TD3 (C-TD3), the first integration of the Twin Delayed Deep Deterministic (TD3) policy gradient algorithm into a chaotic dynamical framework. C-TD3 synergistically combines chaotic neural dynamics modeling with double-delayed deterministic policy optimization, enabling environment-change-triggered adaptive exploration restarts and learning-phase-dependent dynamic modulation of chaotic intensity. Contribution/Results: Empirical evaluation on continuous-control goal-reaching tasks demonstrates that C-TD3 significantly improves policy stability; exploration automatically decays during training yet rapidly reinitiates upon environmental shifts; and optimal performance is achieved at moderate chaotic intensity. This work bridges a critical gap between CBRL and state-of-the-art deep RL algorithms, establishing a novel paradigm for chaos-driven adaptive decision-making.
📝 Abstract
Chaos-based reinforcement learning (CBRL) is a method in which the agent's internal chaotic dynamics drives exploration. This approach offers a model for considering how the biological brain can create variability in its behavior and learn in an exploratory manner. At the same time, it is a learning model that has the ability to automatically switch between exploration and exploitation modes and the potential to realize higher explorations that reflect what it has learned so far. However, the learning algorithms in CBRL have not been well-established in previous studies and have yet to incorporate recent advances in reinforcement learning. This study introduced Twin Delayed Deep Deterministic Policy Gradients (TD3), which is one of the state-of-the-art deep reinforcement learning algorithms that can treat deterministic and continuous action spaces, to CBRL. The validation results provide several insights. First, TD3 works as a learning algorithm for CBRL in a simple goal-reaching task. Second, CBRL agents with TD3 can autonomously suppress their exploratory behavior as learning progresses and resume exploration when the environment changes. Finally, examining the effect of the agent's chaoticity on learning shows that extremely strong chaos negatively impacts the flexible switching between exploration and exploitation.