Frequency and Generalisation of Periodic Activation Functions in Reinforcement Learning

2024-07-09

arXiv.org

AI Summary

This work investigates the frequency evolution and generalization mechanisms of periodic activation functions—such as learnable Fourier features—in deep reinforcement learning. We systematically observe that their parameters consistently converge to high-frequency representations: while improving sample efficiency, this convergence exacerbates overfitting to observation noise, resulting in worse generalization than ReLU. We provide the first empirical evidence that this high-frequency convergence is an intrinsic cause of their generalization deficiency. To address this, we propose weight decay as a synergistic regularizer that effectively suppresses high-frequency overfitting without compromising learning speed, thereby significantly enhancing robustness to observation noise. Our study offers a novel frequency-domain perspective on periodic activations and delivers a practical regularization strategy that jointly optimizes both training efficiency and robust generalization.

Technology Category

• Machine Learning: Reinforcement Learning
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Periodic activation functions, often referred to as learned Fourier features have been widely demonstrated to improve sample efficiency and stability in a variety of deep RL algorithms. Potentially incompatible hypotheses have been made about the source of these improvements. One is that periodic activations learn low frequency representations and as a result avoid overfitting to bootstrapped targets. Another is that periodic activations learn high frequency representations that are more expressive, allowing networks to quickly fit complex value functions. We analyse these claims empirically, finding that periodic representations consistently converge to high frequencies regardless of their initialisation frequency. We also find that while periodic activation functions improve sample efficiency, they exhibit worse generalization on states with added observation noise -- especially when compared to otherwise equivalent networks with ReLU activation functions. Finally, we show that weight decay regularization is able to partially offset the overfitting of periodic activation functions, delivering value functions that learn quickly while also generalizing.

Problem

• Analyzes frequency impact of periodic activation functions in RL.
• Explores generalization issues of periodic activations with noise.
• Investigates weight decay to mitigate overfitting in periodic activations.

Innovation

• Periodic activation functions enhance RL sample efficiency.
• High frequency representations dominate in periodic activations.
• Weight decay mitigates overfitting in periodic activations.