🤖 AI Summary
This work addresses the limitation of fixed, non-adaptive activation functions in neural networks. We propose a learnable piecewise-linear B-spline activation function, jointly optimized end-to-end with network weights to enable dynamic adaptation to task-specific data distributions. Methodologically, we design and systematically evaluate nine bi-level optimization training strategies. Extensive experiments on feedforward neural networks (FNNs) and convolutional neural networks (CNNs) demonstrate substantial improvements: up to 94% reduction in test error for FNNs and 51% for CNNs. The approach significantly enhances both parameter efficiency and predictive accuracy. However, these gains come at the cost of moderately increased training complexity and marginal inference latency. Our study establishes a novel paradigm for structured, learnable activation function modeling and provides an empirical benchmark for future research in adaptive activation design.
📝 Abstract
Activation functions in neural networks are typically selected from a set of empirically validated, commonly used static functions such as ReLU, tanh, or sigmoid. However, by optimizing the shapes of a network's activation functions, we can train models that are more parameter-efficient and accurate by assigning more optimal activations to the neurons. In this paper, I present and compare 9 training methodologies to explore dual-optimization dynamics in neural networks with parameterized linear B-spline activation functions. The experiments realize up to 94% lower end model error rates in FNNs and 51% lower rates in CNNs compared to traditional ReLU-based models. These gains come at the cost of additional development and training complexity as well as end model latency.