A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function

2026-07-03

AI Summary

This work addresses the lack of a unified structural and loss-based interpretation for existing activation functions—such as GELU, ReLU, and SiLU—which hinders the principled design and understanding of novel variants. The authors propose a new perspective grounded in Gaussian-complementary first-order loss, interpreting GELU as a hard linear gating signal with a Gaussian-distributed random threshold. This insight leads to a generalized “threshold–transmission” family of activation functions that explicitly incorporates an adjustable threshold width parameter, accommodating both fixed and learnable uniform-threshold variants. Through probabilistic threshold gating, piecewise polynomial construction, and controlled empirical validation, the proposed activations consistently match or outperform established alternatives across compact vision and language models, while efficiently leveraging a limited transition region.

Abstract

The Gaussian Error Linear Unit is usually motivated as the expected output of an input-dependent stochastic Bernoulli gate. This work gives a complementary interpretation based on the Gaussian complementary first-order loss function: GELU is the signal-transmission term of the expected surplus of a hard linear gate with a Gaussian random threshold. This view separates loss accounting from forward signal transmission and generalises to a threshold-transmission family that includes ReLU, GELU, SiLU/Swish, and hard swish as special cases. The uniform-threshold case recovers a hard-swish-like compact piecewise-polynomial gate with an explicit threshold-width parameter, yielding fixed- and learned-width variants. Controlled experiments on compact vision and language models show that calibrated or learned uniform-threshold gates are consistently competitive with GELU, ReLU, and SiLU/Swish, improve over them in most tested settings, and use the finite transition region nontrivially.

Problem

• GELU
• activation function
• threshold-transmission
• first-order loss
• neural network

Innovation

• threshold-transmission
• GELU
• first-order loss
• uniform-threshold gate
• activation function