Problem
Research questions and friction points this paper is trying to address.
parity problem
linear inseparability
global interactions
neural networks
high-order components
TESLA: Taylor Expansion of Sinusoidal Learnable Activations
🤖 AI Summary
This work addresses the challenge of learning parity functions—a canonical problem hindered by linear non-separability and global dependencies—by introducing TESLA, a novel activation function that explicitly controls polynomial order within the activation layer. Inspired by Fourier series, TESLA employs a learnable combination of sine and cosine components to selectively amplify high-frequency signals, thereby guiding the network to capture global structural patterns. Theoretical guarantees are provided through Lipschitz continuity and Rademacher complexity analyses. Empirically, TESLA achieves strong generalization on 32-dimensional parity tasks using only 100,000 samples—merely 0.002% of the input space—and demonstrates robustness to 30% label noise. It also significantly outperforms baseline methods on Forrelation and ImageNet-100 benchmarks.
📝 Abstract
The parity problem--deciding whether the number of ones in a binary vector is odd or even--remains challenging for standard neural networks due to linear inseparability and the need for global interactions. We propose TESLA, an activation defined as a learnable combination of sine and cosine terms, enabling explicit control over polynomial degree and selective amplification of high-order components. Theoretically, we show that constraining TESLA's coefficients yields Lipschitz/Rademacher complexity bounds and shapes the training dynamics to emphasize higher-frequency structure. Empirically, on parity with input length n = 32, TESLA attains strong generalization with 100K training samples (approximately 0.002% of the 2^32 input space) and remains robust under heavy corruption, retaining high accuracy with up to 30% label noise. We also compare against periodic and frequency-based baselines (SIREN, SNAKE, and Fourier feature embeddings) on parity and Forrelation. Beyond synthetic structure, TESLA delivers comparable performance on ImageNet-100, indicating that activation-level degree control transfers to more general vision workloads. Code: https://github.com/KAU-QuantumAILab/TESLA
Innovation
Methods, ideas, or system contributions that make the work stand out.
learnable activation
Taylor expansion
high-order interactions
frequency amplification
parity problem
🔎 Similar Papers
No similar papers found.
