CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learning
In deep reinforcement learning, exploration suffers from both a lack of theoretical guarantees and suboptimal empirical performance. To address this, we propose a parameter-free exploration mechanism that repurposes the standard critic network—eliminating the need for auxiliary parameters—by modeling state-action value uncertainty as a linear multi-armed bandit and introducing an adaptive scaling strategy alongside a lightweight auxiliary module, CAE+, which adds only ~10 lines of code and <1% extra parameters. Theoretically, our method achieves a sublinear regret bound in continuous state spaces. Empirically, it significantly outperforms state-of-the-art methods on MuJoCo and MiniHack benchmarks. Our approach thus unifies strong theoretical grounding with remarkable engineering simplicity, enabling dual functionality of the critic for both policy evaluation and uncertainty-aware exploration.