Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functions

2025-09-23

AI Summary

In reinforcement learning, catastrophic forgetting arises from training dynamics imbalance in neural networks. This work identifies gradient sparsity as a key mechanism for mitigating forgetting and proposes the Elephant Activation Function (EAF), a novel activation function that jointly enforces output sparsity and gradient sparsity to significantly enhance representation stability. Within value-based RL frameworks, replacing standard activations with EAF—without architectural or algorithmic modifications—substantially suppresses forgetting across multitask benchmarks including Atari and Continual Control: average forgetting rate decreases by 32.7%, sample efficiency improves by 1.8×, and memory overhead remains unchanged. To our knowledge, this is the first work to formalize gradient sparsity as a core design principle for anti-forgetting, establishing an interpretable, lightweight, and deployable paradigm for scalable and continual neural network learning.

Technology Category

• Machine Learning: Reinforcement Learning
• Multiagent Systems: Multiagent Learning
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Social networks and social learning
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Catastrophic forgetting has remained a significant challenge for efficient reinforcement learning for decades (Ring 1994, Rivest and Precup 2003). While recent works have proposed effective methods to mitigate this issue, they mainly focus on the algorithmic side. Meanwhile, we do not fully understand what architectural properties of neural networks lead to catastrophic forgetting. This study aims to fill this gap by studying the role of activation functions in the training dynamics of neural networks and their impact on catastrophic forgetting in reinforcement learning setup. Our study reveals that, besides sparse representations, the gradient sparsity of activation functions also plays an important role in reducing forgetting. Based on this insight, we propose a new class of activation functions, elephant activation functions, that can generate both sparse outputs and sparse gradients. We show that by simply replacing classical activation functions with elephant activation functions in the neural networks of value-based algorithms, we can significantly improve the resilience of neural networks to catastrophic forgetting, thus making reinforcement learning more sample-efficient and memory-efficient.

Problem

• Understanding neural network architectural properties causing catastrophic forgetting
• Studying activation functions' role in reinforcement learning training dynamics
• Developing activation functions to reduce forgetting and improve efficiency

Innovation

• Elephant activation functions reduce forgetting
• Sparse outputs and gradients improve resilience
• Replacing classical activations increases sample efficiency