Z-Plane Neural Networks: Bounded Geometric Activation Replaces ReLU and LayerNorm

2026-06-14

AI Summary

This work addresses the limitations of conventional deep networks that rely on ReLU activations and LayerNorm, which often lead to neuron death, loss of directional information, and disruption of feature orthogonality. Inspired by biological axonal frequency modulation, the authors propose the first deep architecture entirely eliminating ReLU and LayerNorm. The approach models hidden states as bundles of two-dimensional phasors constrained to a hypersphere, employing a radially bounded geometric activation function \( \mathbf{x} / \max(1, \|\mathbf{x}\|_2) \) that caps magnitude while preserving phase. This design ensures isotropy and 1-Lipschitz continuity, enabling stable training of very deep networks without normalization. A 100-layer Z-Plane MLP trained with this method achieves 98.34% accuracy on MNIST, demonstrating that geometric activation alone suffices to guarantee numerical stability and convergence in deep models.

Abstract

Modern deep neural networks rely on Euclidean scalar activations (e.g., ReLU) and global normalization techniques (e.g., LayerNorm) to prevent gradient instability in deep architectures. However, these mechanisms inherently cause dead neurons, discard critical directional information, and destroy the orthogonality of feature representations. Inspired by the frequency-modulation transmission of biological axons, we propose the Z-Plane Neural Network, which maps hidden states into 2D phasor bundles on a hypersphere. We introduce a novel geometric activation function, Radial Bounding($\mathbf{x} / \max(1, \|\mathbf{x}\|_2)$), which limits the energy magnitude while preserving the phase (direction). We demonstrate mathematically that this isotropic activation maintains 1-Lipschitz continuity and prevents gradient vanishing by preserving tangential gradients. Empirically, a 100-layer Z-Plane Multi-Layer Perceptron (MLP)-entirely devoid of ReLU and LayerNorm-successfully converges on the MNIST dataset with 98.34% accuracy and absolute numerical stability, proving that bounded geometric activation alone is sufficient for stable deep learning.

Problem

• dead neurons
• directional information loss
• feature orthogonality destruction
• gradient instability
• Euclidean scalar activations

Innovation

• Z-Plane Neural Network
• geometric activation
• Radial Bounding
• 1-Lipschitz continuity
• phase-preserving representation