Rethinking Neural Nonlinearity as Gating

2026-07-03

AI Summary

This work addresses the lack of a unified formulation among conventional activation functions and their hardware deployment limitations imposed by ADC/DAC bottlenecks. The authors propose Threshold Gating (TG) as a universal primitive for neural nonlinearities, employing an input-conditioned branch gating mechanism that subsumes major activation functions as special cases. Building on this unification, they introduce the “Minimal Branch Theorem,” enabling lossless activation function conversion and seamless model transfer across diverse architectures—such as CNNs, Transformers, and RNNs—without retraining. The framework supports both training from scratch and post-hoc adaptation, offering benefits in model compression, performance enhancement, and training acceleration. Furthermore, it maps efficiently onto analog in-memory computing architectures, substantially reducing power consumption and area overhead.

Abstract

Activation functions are considered an essential primitive for neural nonlinearity, i.e., they enable neural networks to serve as universal approximators. In this paper, we show that this nonlinearity can also be achieved by input-conditioned threshold gating through branches as a universal primitive. We demonstrate that standard activations -- whether piecewise-linear (ReLU, PReLU, Hardtanh) or smooth (SiLU, Sigmoid, Tanh, GELU) -- are in fact instances of a single Threshold Gating (TG) primitive. For softmax, we show that it admits an exact TG conversion via its equivalent per-element Sigmoid form. We then validate these equivalences by converting pretrained networks across CNNs, transformer-based models, and recurrent architectures, preserving model performance without requiring retraining. Threshold Gating also enables training from scratch that goes beyond replacing existing activations, enabling gains in model compression, performance, and shorter training. We also propose a 'Minimal Branch Theorem' which relates the minimum number of required branches in our primitive to the trainability of general deep neural networks. In terms of hardware implementation, TG maps to a unified implementation in the case of analog in-memory systems, addressing the bottleneck of analog-to-digital and digital-to-analog converters (ADC/DAC) that is known to significantly impact power consumption and on-chip area.

Problem

• neural nonlinearity
• activation functions
• threshold gating
• hardware implementation
• analog in-memory computing

Innovation

• Threshold Gating
• Activation Functions
• Neural Nonlinearity
• Analog In-Memory Computing
• Model Compression